In the game industry there is an oft-quoted rule of thumb that states "Creating 80% of the game only takes 20% of the time". It's a variation on the Pareto principle, and in my experience, it is fairly accurate. Even once your game is finished, fully playable, and not to mention, fun, there is still a ton of work to do before you're ready to hit the trigger and let the rest of the world play it.
I've been a key developer on the launch of several major games, even more small ones, and of course the only developer for Tactic's first (and currently, only) game, Immortal Empire. I've seen smooth launches, bumpy launches, and delicious lunches. Let me make this next point abundantly clear by writing it in the largest, boldest font this website will allow:
Having a smooth launch for your indie game is extremely important.
Your dev-instincts have probably already alerted you to this, but let me just drive it home with a few important factoids and common misunderstandings.
- This rule does not really apply to major studios. If you want to point to Skyrim being buggy, or World of Warcraft's bumpy launch despite their respective successes, please understand that this doesn't apply to you. This is because:
- People will not wait for your game to work. You will get a big boost to traffic when your game launches. If your game is not working for a particular user, that user will very likely never ever play your game again. Boost to traffic wasted. And it's not just the users:
- Distribution platforms measure you based on a very brief period during your initial launch. Platforms like Steam or Kongregate will decide how much advertising your game will subsequently receive based on how it performs at launch. At launch. Not 2 weeks after launch, and certainly not 6 months later even if you significantly update or improve your game. And finally:
- Game reviews will come at game launch. Even getting someone to review your game can be difficult. Getting them to re-review it after you've fixed all the bugs? Forget it. And expect a bad review.
If the above doesn't make it clear - don't don't don't launch a half-finished or even 90% finished product with the idea that you will "continually update it" as it grows into a massive success. Finish your game, then launch it. If it's a success, update it and grow your community.
One key clarification. When I say launch - I'm referring to your "big launch". For example, when you go live on Steam to the general public. Closed betas, alpha funding, and other soft launch techniques are actually something I'm about to recommend.
So you're launching a massively multiplayer game and don't exactly have thousands of trained quality assurance testers to make sure it's working great. How do you make sure you have a smooth launch? It's a mix of prevention and damage control.
Tip 1 - Have a fast build / deployment system
Your bug-free game has lots of bugs. Sorry :(. Don't believe me? The most recent patch for Starcraft 1 was released in 2009, over 10 years after its original release date. Fixing a typo? Nope - just little things like being able to nuke from anywhere on the map.
 |
| Sometimes your patch might even introduce bugs! |
You're going to need to patch your game, so make sure that you have a fast, 1-touch build and deployment system ready so you can catch anything that slips through the cracks immediately.
I've seen people try to follow a complex 20-step process just to build and package up their game for deployment, and the whole time they are quadruple checking each step because they don't want to make a mistake. Don't do this! Automate your build and deployment process. Write a program, a script, anything! Throw some (preferably lots of) automated testing in there too! It is much faster, safer, and gives you confidence that your game has been deployed properly.
Most importantly, it means you can very rapidly fix bugs as soon as they are discovered. Which brings me to:
Tip 2 - Have an automated bug reporting system
So we've established that your game will have bugs, despite your best efforts. The vast majority of people will never tell you that they have encountered a bug. During the launch window, if it's a severe bug, or your server has crashed, they will most likely just silently quit your game and never play it again.
Use asserts, detect crashes, keep a useful running log of user actions on the client machine to aid debugging and upload reports to your web server.
Tip 3 - Implement real-time memory and performance profiling
Some bugs are simple and easy to reproduce. Others are more insidious. Your game might be slowly leaking memory or fragmenting. Your performance might be worsening over time or spiking in certain situations you hadn't previously encountered. Not only is it useful to have in-engine real-time profiling tools for development reasons, it is very useful for catching these sorts of bugs and fixing them.
Keep track of poor performance or unusually high memory and report such cases using your automated bug reporting system.
It's also important to check these things during development. So you should:
Tip 4 - Run automated tests during development
 |
| "yo jsut hire a bonch of roobots to test ur game11`" - xXgamedevkilla16Xx |
You should be running the 3 main "S"'s of testing: Smoke, Soak, and Stress tests.
- The type of smoke tests that I recommend are basically a barrage of unit tests. Check to see if everything works at a basic level.
- Everything from "Does the game run?" to "When I execute a trade with another player, do they end up with the expected item?"
- Usually a programmer will manually test the code they just wrote for its direct functionality. Smoke tests are excellent for catching situations where the code you wrote has inadvertently broken other areas of code.
- (PS - proper code architecture can minimize this, but that's for a future blog post.)
- Soak tests will generally run the game for extended periods of time looking for memory bloats, leaks, performance leaks and such.
- Boot the game, and leave it running executing simple "normal" actions for long periods of time (say, 24 hours. Weeks if it is a game server.) For example, log a player in. Say some things in chat. Enter a game room. Kill a monster. Quit. Repeat!
- Boot the game and load all your levels in a loop over and over again. This might reveal leaks caused by level transitions.
- Boot a level and warp to various areas in the game, (in particular for 3d games) looking for performance leaks. Boot next level, and repeat.
- (PS - even garbage collected languages or engines can experience memory bloats. You're not safe! Run soak tests!)
- Stress tests attempt to maximize a particular element of your game. This will give you a good read on the capacity of your game and reveal problems that otherwise would remain hidden until an end user experiences it.
- Fill a level with highly intensive graphics to stress the GPU. Use realistic situations - the goal isn't just to overheat your video card. Spawn more enemies, have them all firing weapons and blasting particle systems, and so forth.
- Fill a level with highly intensive computations to stress the CPU. Again, use real-world situations to test your game, we're not here to just eat up cpu cycles. Run complex physics simulations with thousands of entities. Execute huge numbers of costly pathing lookups for units in the game.
- Fire large numbers of database accesses, or login large numbers of users to your multiplayer game, flood bandwidth with large amounts of network traffic (or, better yet, limit your bandwidth to see how your game performs under poor network conditions).
You might think the cases you're developing will never happen and aren't worth testing. This is exactly the fallacy that causes bugs to slip through the cracks.
Run these tests regularly. Daily, or with every build. Some companies will run this on every changelist that's submitted. Running tests frequently lets you isolate what changes you made that may have contributed to degradation of memory, performance, and most important, stability.
These tests also help identify bottlenecks. At one point when we were developing Bioshock 2 multiplayer, if every player equipped the incinerate plasmid and blasted each other, performance would grind to a halt. We found this case using a stress test and optimized that plasmid until it performed much better.
When I first ran stress tests on Immortal Empire I saw very quickly how many concurrent users the server could potentially handle, and found that database accesses and unit pathing were bottlenecks. So, I optimized all the DB accesses and offloaded those calculations to a separate server. I also severely optimized the pathing to handle common failure cases (where no path was found) as they were the most costly operations.
Tip 5 - Soft launch your game. Repeatedly.
 |
| Houston, we have a pillow. |
Other than QA and automated testing, your best defense against a bad "hard launch" is to soft launch your game as much as possible. The idea is simple: launch your game to a select group of people who are aware your game is a work in progress. People are going to do things your automated tests aren't.
Even with all the automated testing, Immortal Empire did the following and found bugs at every step of the way.
- Closed beta for friends and family (~50 people)
- Closed beta for select users (~300 people)
- Open beta with small press release (~2000 people)
- Soft launch Kongregate (~10000 people)
- "Hard launch" Kongregate (~100000 people)
- Steam (Still to come!)
Friends, family and select users will be vastly more forgiving than the average public. For us, the "open beta" and "soft launch Kongregate" included random users and let us gather game statistics too - what immortals were most popular? The most powerful? What spells seem to be dealing too much damage or too little? Which items are people choosing? Are they too strong? This type of data mining helps tremendously to balance your game and make it more fun and fair for everyone.
The remaining tips pertain only to multiplayer games. I have released many many real-time multiplayer games in my career and can confidently say they have their own set of unique, much more difficult problems than single player games.
Tip 6 - Schedule maintenance with a shutdown warning for connected users
Don't just tear down your server if you've found an exploit or crash. Schedule maintenance, allow it to continue, and start fixing the problem. When the maintenance window comes around, all you'll have to do is use your fast deployment system and boom! You're back in action.
Players don't like their game to be down for extended periods of time, but if there has to be a shutdown, they are far more forgiving when it's scheduled in advance. This gives them confidence that the shutdown was planned and that everything will be running again soon.
When it is time to shutdown your server, don't just disconnect players in the middle of a game. Give ample warning that the server is about to go down, and disable creation of new games. Wait until most of the games are finished and then tear it all down.
Tip 7 - Use separate staging and release game servers
This is the most optional tip in this list, but it's pretty handy as it will allow you to patch your game server with no perceptible downtime to your end user.
For you programmery types this is double buffering for game servers.
Basically you have 2 sets of servers running concurrently, one for staging and one for release. No one is logged onto your staging server, only the release server. It's time to patch! Update your staging server and leave your release server running. Take as much time as you need, because no one is logged on. Once it's ready to go, redirect all new traffic to the staging server.
Then, as players finish up their games and log off the release server, the total number of users will eventually reach zero. Now, switch them! Your staging server becomes the release, and the old release becomes staging. Voila! You just patched your game code and no one had to endure any downtime.
Tip 8 - Put a hard limit on the total number of concurrent users
Believe it or not, people actually will happily wait in line for something. But if you let them in and realize it's too full and ask them to leave, that's not going to go over well.
It's true that you want to have the capacity to support unlimited number of people. But you can't reasonably expect to estimate how popular your game will become, and server costs can be expensive, so overcompensating is unwise. It's better to tell someone "sorry wait in line" than have them overwhelm the server causing it to screech to a halt.
Use your stress tests to get a sense of your server capacity for network bandwidth, CPU consumption, and memory. Then, cap the number of users to ~80% of that limit. Finally, use your real-time memory and performance profiling to measure live users to get a sense of whether your estimated capacity is accurate. Code in support to adjust that capacity real-time as needed.
Traffic for a new game is typically very spiky. On Immortal Empire we were running a small open beta and averaging ~20 concurrent users. Overnight, we had a favourable news article appear on Destructoid. I woke up to ~200 concurrent users and a slew of bugs.
Fortunately, I had prior experience launching multiplayer games and was ready! I followed the above tips and the server kept running despite the unexpected spike in traffic during our soft launch. Then I scheduled some maintenance and released a patch a few days later.
Conclusion
So the million dollar question: Is it worth it? I'll be the first to admit that if you're writing a small, single player Tetris-clone, you probably don't need to implement anything on this list. I've released many small scale games that didn't have these systems in place and they launched fine.
But as the complexity of your game increases, the above tips rapidly become requirements. I've been to many major game studios, and these systems are commonplace. For any medium scale or greater game, or any multiplayer game with centralized servers, you're going to want this stuff.
You put so much effort into your game, you'd be throwing all that effort away if you aren't properly prepared.
Thanks for reading, and I hope this helps you have a smooooooooooooth peanut butter lunch! I mean launch.
Addendum
Immortal Empire was greenlit on Steam back in July 2014 and we're gearing up for another "hard launch". We're currently running a Kickstarter to help fund some significant improvements to the game before we go live on Steam. A stretch goal on that Kickstarter is to release our game engine and source code for free.
So if you'd like to see how we implemented the above features, or simply want to use our game engine (for free!) to make your own game (and never have to build any of the above systems) then mosey on down to our Kickstarter and give it a look-see. Thanks!
