Postmortem: Asheron’s Call

By Ellen Ripley, May 25, 2000

Postmortem: Turbine Entertainment’s Asheron’s Call
By Toby Ragaini 
Gamasutra
May 25, 2000

Asheron’s Call is a statistical anomaly. In an industry where cancelled games and dashed hopes are the norm, this project seemed one day away from certain failure for nearly its entire history. And yet, thanks to the visionary foresight of a handful of people, a healthy dose of luck, and incredible conviction from both the development team and publisher, it made it to store shelves and has received a great deal of critical acclaim.

In May 1995, I walked into a small suburban home in southern Massachusetts and met my new co-workers. Having left my previous job at a genetics lab, I expected nothing more than an interesting summer project as “A Game Writer.” Little did I realize what was in store for me and this start-up company called Turbine.

Having filled every nook of a residential home with PCs, an enterprising group of about ten developers was already busy working on the game that would one day become Asheron’s Call. Although not a single one of them had professional game development experience, I was immediately impressed with their enthusiasm and dedication. After introductions, I was told to scrounge around for a desk. Upon securing an end table and a plastic lawn chair, I sat down and started meeting with various team members to figure out just what this game was all about.

What was described to me was something that nearly every computer game geek is by now familiar with: a 3D graphical MUD. A persistent fantasy environment where hundreds of players could explore the land, defeat monsters, form adventuring parties, delve into dungeons, and complete quests. I’m not sure why anyone thought it was possible. We had no office, no technology to speak of, and no publisher. And I was being paid $800 a month. Yet from these humble beginnings, something truly wonderful was created.

The development team was divided into functional departments. Tim Brennen, a Brown University dropout who had helped develop Windows NT as a Microsoft intern, led the engineering team and would go on to design the server, networking, and character database. Chris Dyl, a former physicist turned programmer, would develop the 3D graphics engine and server-side physics. Andy Reiff, also a Brown alumnus, would later round out the engineering leads as the game systems programmer, responsible for implementing all of the game rules systems and functional interactions in the game world. All of the game’s code would be developed from scratch. At the time, this was a fairly easy decision, since licensable game code was pretty much nonexistent in 1995.

On the art team, Jason Booth, a music student with experience using Lightwave, would take on the title of lead technical artist. In this role, Jason bridged the gap between the art and graphics teams, ensuring that the art asset pipeline ran smoothly. Sean Huxter brought his substantial animation and modeling experience to the team as the lead artist.

Dynamic load balancing on the server gave Asheron’s Callan expansive, seamless game world that required no load times.

My own contributions to the team were in the area of game design. As the project grew in scope, my role changed to become that of lead designer. Soon realizing the amount of work required to design a game with the scope of Asheron’s Call, I put together a team of designers that envisioned and documented the characters, monsters, history, and timeline of a fantasy world called Dereth. In addition, the design team spec’d all of the game rules and systems necessary to RPGs.

Although the team had no professional game development experience, one invaluable thing that the team did have was experience playing MUDs and similar text-based Internet games. Although these games were comparatively simple, the game-play dynamics created in a massively-multiplayer environment are extremely different from a single-player game. MUDs proved to be a very useful model for multiplayer gaming patterns.

Asheron’s Call was initially designed to support just 200 simultaneous players, each paying an hourly fee. Turbine would host the servers, which were originally going to be PCs running Linux. Although in today’s market, this sounds ludicrous, in 1995 this was in fact the standard premium online game model. Games using similar models, like Genie’s Cyberstrike and America Online’s Neverwinter Nights, were quite successful at the time. Based on this goal, the original schedule had Asheron’s Call shipping in the fourth quarter of 1997.

What Went Right

1. Staying true to our original vision of the game.

Asheron’s Call was a ridiculously ambitious project for an unproven team. Yet despite this naïveté (or more likely because of it), the final product is frighteningly close to the original goal of the project. Of course during that time, Turbine learned lessons in feature cutting, scheduling risks, and compromise. But despite all the missed deadlines, all-nighters, and other disappointments, we are able look back on our shared vision and take pride in that we achieved what we set out to do.

Typically, there exists a master document that describes the overall game concept and goal. Although the documentation at the inception of the game was in fact very sparse, what little that did exist described the fundamental architecture of the game, including its client/server model, dynamic load balancing capabilities (described later), and 3D graphics. In addition, game-play details such as the allegiance system, magic economy, and the emphasis on social game play are in my notes going as far back as 1995. The team internalized these goals, and a form of oral tradition maintained them in meetings.

Shot of World Builder decorating an Empyrean ruin on the landscape in the desert. The pillars on that building are hand-placed, as was the portal and the lights that stand outside the door.

Although we didn’t know it at the time, Asheron’s Call would debut as the third massively-multiplayer online RPG amidst two strong competitors, Ultima Online and Everquest. We’re often asked if we made any dramatic changes in response to the release of these two titles. In all honesty, the answer is no. If anything, these two products proved to us that our initial technical and game design decisions were correct. Clearly, social game play helped drive the success of these games. This made our game’s social systems such as allegiance and fellowships all the more important. It was also obvious that immersion was critical. Instability and pauses were the bane of massively-multiplayer games. In theory, the dynamically load-balanced servers would prevent many of these problems.

In an industry that can be driven by holiday deadlines, marketing hype, and cutting corners, it’s refreshing to know that ambitious goals can still be rewarded. But it’s more than that. While we certainly could have created a less ambitious game, I believe it would have been a detriment to Turbine’s competitiveness as an independent development studio. Asheron’s Call might have shipped earlier had it been a LAN game or a series of connected arenas, but we would not have the innovative technology and game design experience that today puts Turbine in such a desirable competitive position in the industry. In this way, our team’s unwavering vision was handsomely rewarded.

2. Securing a publishing agreement with Microsoft.

In mid-1996, representatives from the newly-formed MSN Gaming Zone were booed by the audience of the first Mpath Developer Conference. Their crime was the prediction that hourly fees were dead and that flat monthly rates would become standard. Our business plan at the time counted on an hourly model, but we recognized the truth to the Zone team’s statement. At that year’s E3, we relentlessly pursued Jon Grande, product planner on the Zone, in order to pitch him our game proposal and show him our technology demo. At that time, the demo consisted of two PCs connected to each other. One was running the client software, complete with 3D graphics. The other was the server executable. The Zone team was very impressed, and scheduled a visit to our office (we’d since moved into an actual office space south of Boston). Soon after the visit, Microsoft agreed to enter into a publishing agreement with Turbine, secured initially with a letter of intent. The actual contract arrived six months later, but the letter of intent granted us an initial milestone payment and enough certainty to schedule the milestone deliverables. This was the start of a long, sometimes tumultuous, but ultimately fruitful alliance.

Shot of the in-house dungeon creation tool. Once the dungeon is laid out the decorating can begin. The panel to the right is a list of hand-placed objects in the current piece. Note the mine wheelbarrows and picks on the wall.

After we secured the contract, the division of labor was discussed. As the developer, Turbine was to design the game, engineer and implement all of the code, generate all art assets, create a QA plan, and perform testing on all game content. With its pre-existing Zone platform, Microsoft was responsible for code testing, billing, and ongoing server operations. Fundamentally, this meant that while Turbine would create the game, the day-to-day operations of the Asheron’s Call service would be entrusted to Microsoft.

One thing that Turbine successfully negotiated for was the rights to our source code. Besides the team, we knew that our massively-multiplayer technology was going to be our single most valuable asset. In addition, we agreed to a one-title deal that gave us the flexibility to pursue other development deals as opportunities arose. In this way, we ensured that Turbine would remain independent and effectively in control of our own destiny.

In many respects, Microsoft proved to be an ideal partner for Turbine. Like Turbine, the Zone was a start-up organization, and was eager to prove itself. The Zone was pioneering a new type of business, with a business model new to Microsoft, and this placed the managers of the Zone in a position where they could afford to take risks. And while Asheron’s Call ultimately validated Microsoft’s belief in Turbine, at that point Turbine was certainly a risk.

Besides the obvious funding issue, Turbine benefitted from its partnership with Microsoft in other ways. We had free access to Microsoft development tools like Visual C++, Visual SourceSafe, and a bug-tracking database called RAID. We learned a lot about professional software development from Microsoft as well, such how to create an efficient build process, manage code source trees, and organize effective test cycles on the daily builds. Finally, we gained prestige by working with one of the most respected software companies in the world. Having Microsoft as a partner gave us a lot of credibility and put us in a much better position to pursue funding and make critical hires, two incredibly important objectives for a small start-up company.

3. Reusable engine and tools.

Massively-multiplayer games require a fundamentally different architecture from that of single-player games, or even multiplayer LAN games. Beyond the graphics engine, user interface, and other elements of a typical game, persistent massively-multiplayer games generally require a centralized server, networking layer, user authentication, game administration tools, and a host of other technologies.

Early on, Turbine recognized that many of these technologies would be required by any massively multiplayer game, and could perhaps be generalized enough that they could be reused in different massively-multiplayer titles. At the time, this was an unusual premise for a game developer; typically, source code was thrown out at the end of a project, and the idea of licensing a 3D engine like Quakewas still a long way off. From our perspective it just made good business sense to leverage our R&D as much as possible. Since so much of our development budget was devoted to creating these key technologies, we made every effort to keep the technology modular and data-independent.

Shot of the landscape test program. Designers ran around the landscape off-line to see what had been added and how it looked. This is a shot of the obsidian plane with mushrooms and crystals, not to mention some creatures lurking near a mysterious power source.

This modular architecture has since proven to be a tremendous win for Turbine. We’ve been able to prototype new game concepts rapidly by changing data while keeping the server executable nearly unchanged. Not only has this helped us get new business, it has also proven to be extremely useful for in-house play testing and constructing proof-of-concept demos.

Currently we are investigating the potential of licensing our technology. While we continue to advance the code base, we have placed some emphasis on productizing the Turbine engine. From a business perspective, this is a very desirable source of revenue. We can leverage our R&D efforts and development costs, while advancing the engine that our own future products will use.

In addition to the ability to reuse code, Turbine’s modular emphasis extended to the way content is created for the game world. As development on Asheron’s Call progressed, we quickly came to realize that populating a game world the size of Dereth was going to be a monumental task. By this time, we knew our competitors were hiring teams to design individual levels and create content manually. This seemed less than optimal to us, and furthermore we didn’t have the resources to hire a large content team.

Instead, we created a series of world-building tools to maximize our efforts. The first kind of tool allowed artists to create vast chunks of game environment (represented as a grayscale height map) with each stroke of their brush. Random monster encounters and terrain features such as trees and butterflies could also be placed using this method.

We also developed a tool called Dungeon Maker to create subterranean environments such as dungeons and catacombs. Early on, Jason Booth got sick of hand-modeling the complex level designs he was getting from the design team, so he and user-interface programmer Mike Ferrier created a level-building tool that used an intuitive drag-and-drop interface. This allowed nontechnical designers the ability to create and instantiate dungeons quickly without taking up the art team’s valuable time.

Different types of combat were developed independently, causing complications.

An offshoot of Dungeon Maker, World Builder, became a much more advanced tool by the time Asheron’s Call shipped. Using World Builder, a content designer could wander around the game world placing houses, decorations, and monster encounters, and even raise and lower the terrain. This proved to be an incredible time-saver, and the amount of landscape content we were able to generate easily quadrupled.

This kind of tool modularity allowed us the ability to update the game world easily with new content, such as new monsters, quests, items, and adventure locations. Thanks to monthly content additions, Asheron’s Call “events” can propel an overarching story forward and involves players in all areas of the games. So far these events have proved to be a huge success. Players feel like they are part of a living, breathing world, and are more likely to stay involved in the game for longer periods of time.

4. Painless launch.

When the first few thousand players began pouring onto the production servers, we were certain that there would be all sorts of catastrophes. We had watched our competitors suffer similar calamities, and we had resigned ourselves to accept this rite of passage. To our surprise, nothing went wrong the first day. We were delighted by just how stable and uneventful the retail launch was. Everything went without a hitch.

Each Asheron’s Call server had to support 3,000 concurrent players, minimum.

This stability was due to effective beta testing, intelligent project management, and insightful data-center equipment deployment. Here’s how it worked. During beta, both Microsoft and Turbine testers submitted bugs into RAID. In addition, user-submitted bugs were tracked by the Microsoft team and were added into RAID if they were deemed important. Server performance metrics were one of the key goals towards meeting our shipping requirements. Each server had to maintain a minimum level of performance, given a concurrent user base of 3,000 players. To meet this metric, a few changes were in order. The server-side physics was modified to use a more simplified collision model. In addition, a faster “clean-up” cycle for objects dropped on the landscape was implemented. Having made these changes, we were able to meet the aggressive server metrics and our server software has since proved to be nearly bulletproof. In fact, for the first several weeks, the server software did not crash once, which was a major accomplishment considering the technical problems evident in other massively-multiplayer games.

Our retail launch was a staggered affair. Initially, only two “enthusiast-oriented” retail chains received shipments of Asheron’s Call boxes. This allowed our die-hard fans from the beta testing program to get copies, but prevented the deluge that would have occurred had we been in the larger, more mainstream retail stores. While it would have been exciting to see massive sales on day one, I believe that this gradual approach was a smart move.

5. Seamless environment using dynamically load-balancing servers.

One the most impressive features of the Turbine engine is the continuous outdoor environment. This is made possible thanks to dynamic load balancing, which is a scalable server-side architecture. The easiest way to appreciate the need for dynamic load balancing is to consider the following scenario.

Imagine a hypothetical game world that is divided into four servers, each of which corresponds to a geographic area in the game world. With a static server architecture, if everyone in the game world decides to go to the same area, that one server’s performance would be dramatically impaired, while the three remaining servers would effectively be idle, completely unaware of their overtaxed brother.

WorldBuilder let Turbine place monster encounters easily.

Dynamic load balancing solves this overloaded server problem. Instead of assigning a static geographic area to each server, the individual servers can divide up the game world based on the relative processor load of each server. In the previous example, instead of remaining idle, all four servers would divide the load equally among themselves, ensuring the most efficient use of the hardware’s processing capacity.

Dynamic load balancing allows a very free-form environment where players can travel wherever they want with very few hard-coded limits. But in order for the graphics engine to accommodate the seamless nature of the server, we couldn’t allow a “level loading” pause typical in many 3D games to interrupt the game play. To avoid level-loading, the geometry team headed by Chris Dyl engineered a unique rendering engine that constantly loads data in the background, and draws objects at far enough distances so as to minimize obvious “popping” effects and without having to rely on a fogging effect to hide the clipping plane.

What Went Wrong

1. Poor scheduling and communication.

For most of its early history, Asheron’s Call was the victim of poor project management. During the last year of development, a management reorganization took place that salvaged the project. Depending on how far back you look at the schedules, Asheron’s Call was either one to two years late. This is attributable to a number of reasons, some of which I will explain momentarily.

When Microsoft and Turbine entered into the development agreement, neither side had any idea of the scope of the project. An initial list of milestones was drawn up by the Microsoft product manager and our development leads. Unfortunately, after the second milestone, deadlines were consistently missed. A lot of this was due simply to underestimating the time required for development tasks. This created a domino effect as we continually played catch-up, trying desperately to make up for lost time.

This is a shot of the Banderling being animated. He is performing a large swing to the front, and is caught in mid-backswing. The weapon is rooted out of the scene when the animation is preprocessed, as is the floor. Note the movement path of the weapon as it strikes.

This schedule free-fall continued into 1997 and forced us to re-evaluate the feature set. Unfortunately, feature cuts were made without considering the impact on the playability of the game. Ultimately, most of these features were added back into the game anyway, which took additional time due to the reallocation of team resources. The lesson here concerns the value of effective scheduling. Identify the risky areas in your schedule early, figure out the dependencies, and make sure you pad the time estimates for tasks.

Communication between Microsoft and Turbine was also a major factor. The teams were separated by about 3,000 miles and three time zones. Although weekly conference calls were scheduled, they lacked the collaborative mentality necessary for maintaining a successful relationship. E-mail threads were either ignored or else escalated into tense phone calls, and in some cases the bug-tracking database (RAID) was not used effectively.

Clearly, everyone would have benefited from more face-to-face time. E-mail – and even conference calls – are poor media for managing new and sensitive corporate relationships, especially ones between companies with such different corporate cultures. From a developer’s perspective, it’s always easy to blame the publisher for unrealistic expectations and bureaucracy. What’s important to realize is that it is everyone’s obligation to communicate expectations and problems before they escalate to the point of being a crisis.

2. Inexperienced development team.

None of the senior developers at Turbine (including me) had ever shipped a retail PC game. None. Many of the employees were students immediately out of college, or even college students completing a work-study program. This obviously was the source of several severe problems in the development of Asheron’s Call.

This is a shot of the Banderling being animated. The motion graph shows the keyframes and the spline movement of the Banderling’s chest as he makes a swing with a weapon.

It was nearly impossible for team leads to give realistic schedule estimates for tasks, since few of us had experience in professional software development. It was also initially difficult to get different teams from the programming, art, and design departments to communicate regularly with each other. The collegiate atmosphere made it very difficult for decisions to be made; meetings would happen and resolutions would seemingly be agreed upon, only to have those same questions asked in a subsequent meeting. No one likes unnecessary bureaucracy and giving up creative freedom, but ultimately one person needs to be given the authority to make a decision and hold people to it. A good supervisor takes into account the opinions of everyone involved; design by committee simply does not work.

Obviously, having a seasoned and experienced development team has innumerable advantages. While it’s not critical that everyone on the development team have professional experience, at the very least team leads should have some form of professional experience. As it was, Turbine had to get by with raw talent, unabashed enthusiasm, and simply not knowing any better.

3. No feature iteration during development.

Many weaknesses of Asheron’s Call at launch stemmed from the methodology we followed for feature completion. Features were scheduled by milestone and were expected to be completed in their entirety before other features were worked on. While this approach may work for more typical software applications, PC games rely on a host of interrelating systems that cannot be implemented in a vacuum.

Early shot of a mock-up UI that had very limited functionality. Some of the buttons worked, but mainly this was a prototype pasted over the 3D window, which was working at the time. The upper-left set of buttons were macros. The paper doll was hand drawn, and did not reflect the look of the character.

An example of this involved our melee combat system. This game feature was completely spec’d and implemented long before magic spells worked within the game, under the misguided assumption that it saves developer and test resources not to have to revisit completed features. Clearly, these two game systems needed to be tested and balanced in stages alongside each other, not independently.

Another example of this problem occurred during beta testing. A massively-multiplayer game cannot be considered adequately tested until thousands of players have participated in the game world for at least a few months. The first time Asheron’s Call was exposed to this many users was when it went into beta testing. Unfortunately, we were placed in a code freeze situation during the beta test, and only the most serious bugs were fixed.

Both Microsoft and Turbine recognized many serious game balancing problems during beta, but at that point it was extremely difficult to make changes. This can be attributed to our tight schedule, but earlier beta tests would have accelerated the bug-finding process and resulted in a better balanced game. On future projects, Turbine is deploying a more iterative implementation process where rapid prototyping and early play-testing is encouraged.

4. An ambitious project lacking fundamental underlying technologies.

As one of the first massively-multiplayer 3D games, Asheron’s Call was a bold undertaking. Several core components were still theoretical when the project was planned. Things like dynamically load-balanced servers and continuous, uninterrupted outdoor environments were still unproven concepts when we committed to them for Asheron’s Call. Furthermore, we had to create our own 3D graphics engine, a latency-friendly network layer, and physics and game rule systems that would all work within a client/server model.

This is an incredibly early shot of our Character Generator screen. Much of the interface has remained unchanged, but the same cannot be said of the faces. This was probably version one of the face images. Turbine has gone through a couple of versions since then.

We learned very quickly why there hadn’t been a game like Asheron’s Call before us: It was damned hard to develop such a game. I don’t think committing to a less aggressive feature set was the right solution, though. Instead, we should have acknowledged up front that R&D efforts are fundamentally hard to schedule, and been more flexible with our development schedule. With this in mind, we could have created more realistic estimates and done a better job managing expectations within and outside Turbine.

5.No documented high-level feature statement.

Because Asheron’s Call had such a long and evolving development cycle, it was difficult to keep all the documentation up-to-date. To compound the matter, the project never had an official feature set as part of the development contract with Microsoft. The technical design document process and high-level feature overviews were basically skipped. This created severe problems when it came to prioritizing which features were important. We constantly had to justify features, and we had no documentation to fall back on to resolve our discussions.

It was difficult to play-balance the game since that aspect of development took place after the code was frozen

Without a high-level vision statement it was also very difficult to educate new employees about the game. There was a sort of oral tradition to initiate new employees that had been passed down for so long that it just became part of our company’s culture. This was partially possible because the concept of a 3D graphical MUD intuitively made sense to a lot of people. Unfortunately, it was very difficult to explain what Asheron’s Call was about to people who didn’t understand this concept or had their own ideas about how things should be done. Having a documented vision statement and a description of the high-level feature set is absolutely essential for any title.

A Unique Company Résumé

Asheron’s Call was a tremendous learning opportunity for Turbine and Microsoft. Despite all the problems and setbacks, Asheron’s Call is a success story. The game has been well received by PC game enthusiasts as well as the majority of the game industry press. The fan support for Asheron’s Call is overwhelming, and players routinely spend more than six hours a day logged into the game world.

In addition, Turbine is now in a very desirable position, being one of only a handful of developers (and the only independent studio) that has successfully created a massively-multiplayer title. Industry analysts predict that online games will be the fastest growing segment of entertainment software. With its reusable architecture, robust toolset, and (now) experienced developers, Turbine intends to remain at the forefront of massively-multiplayer gaming.

Toby Ragaini leads Turbine Entertainment Software’s design team and was the lead designer of Asheron’s Call. He is currently working on Turbine’s next-generation massively-multiplayer game.

Copyright © 2003 CMP Media Inc. All rights reserved.