Thursday 8 October 2015

Effective Coding: Ya Ain't Gonna Need It

This is the second instalment of a series of articles featuring an in-depth discussion of programming principles, as they pertain to Video Game programming.  The topic this time is a very important principle - "Ya Ain't Gonna Need It" or YAGNI.

If you find you learn better via video - check out the video - where my facial expression captured in the auto generated thumbnail turned out pretty hilarious - below.




What is YAGNI?

This principle comes into play when you're programming something that you're guessing you might need at some point in the future. The basic premise is that generally speaking, you aren't going to need it, and time spent writing this code was a waste of effort.

On the whole, this is a time management principle, and one that I find comes up commonly in games, a field of programming that frequently suffers from the project management holy trinity of feature creep, competition anxiety, and short deadlines.

This actually happens?

Like many of these principles, it sounds obvious. Why would you ever write something that you might never use? But in practice, this is a very common problem that can cause enormous inefficiencies and can be more difficult to solve than you might initially think.

Further, it is sometimes difficult to identify when you're violating this principle. If you're busy slaving away at your desk while some other programmer is spending half the day playing ping pong and eating office snacks, you clearly feel like the more efficient employee.  However if the code you are writing will never be used, then sadly, that slacking programmer is potentially vastly more productive than you.

There are two ways this problem usually manifests. One is simple to resolve, the other is more difficult.

The easy case

The more obvious case is when you are developing a new feature that you don't really have a current use for.  You're doing this on the presumption that you might use it at some point in the future.

For example, say you're busy making a first person game. You read some article with a great technique on how to implement a third person camera, and you decide to implement it despite that your game doesn't really need this functionality.

You'll start to justify your decision.
  • It will probably be useful for debugging
  • It will only take me an hour
  • The article is fresh in my mind, and I was just working in the camera code recently, so it's faster if I do it now
  • If we ever need a third person camera, then it will be already ready and waiting
  • It's a good learning experience
  • I'm excited to work on it, and it will be fun

They all sound like good reasons! Quite convincing really! And lo you will have just talked yourself into what's most likely going to be a waste of time and effort.

Why is it a waste of time?

Only the last two reasons from the above list are valid. Writing code for fun and education is an excellent pastime that I thoroughly endorse. You might even argue that writing some code for fun is worthwhile as an occasional sojourn that reinvigorates your spirits and boosts productivity in the long run. Of course, do this too much and a looming deadline will begin to disagree with you.

The other reasons, the ones you likely told your project manager, are patently bullshit. I've seen projects become extremely derailed in situations where unchecked programmers have been spinning their wheels with seemingly zero friction.

What's wrong with the other reasons?
  • You're taking a gamble that it might be useful
  • Re-familiarizing yourself with the article won't take long. It will be just as fast to implement this later when (and if) it's needed
  • Be realistic, it's definitely going to take longer than an hour
In addition to the time sink,
  • You will have increased the complexity of your code
  • You have potentially introduced bugs
  • There is now more code to maintain in the future
Solution to the easy case?

I said it was easy - and it is. Just defer working on unnecessary features until you actually need them.

Let's move on to the more difficult case.

The hard case

The harder case is when you're working on a feature that you know you need, but it's not clear how much of the logic should be hardcoded versus architecting some more robust system that affords huge customizability for the game designer.

It is typically more time consuming to write a highly architected system than hard-coding some basic logic, but there are cases where architecture provides a much more graceful solution and gives a net benefit to efficiency.  So how do you decide what to do?

Obviously each game is different and there is no single solution for all situations. Nonetheless, this article is about overarching principles, and YAGNI gives us a solid ground rule to follow: Start with hardcoding, and scale up from there.

Let's take a look at some examples to see why this is effective.

Examples

The most common area of game code where I encounter this problem is Artificial Intelligence. Having seen many AI systems for a variety of games big and small, I've encountered varying levels of robustness producing similarly varying degrees of success. 

When I say Artificial Intelligence I'm referring to behaviour of enemies, NPCs and the like. Consider this example archetype from a cover shooter:

The enemy runs to a cover point in range of the player, takes cover, shoots periodically, and retreats to a cover point farther away if the player gets too close.

You can break this into two chunks.
  • Actions - Lower level systems that are often shared across enemies and involve no decision making: movement, taking cover, shooting
  • Behaviours - Higher level sytems that are often unique across enemies and are decision focused: When do I shoot? When do I run? When do I take cover?
The actions are simple. These are usually hardcoded with certain variables customizable by a game designer such as movespeed or attack range.

Behaviours

Behaviours are where things start to get fuzzy. You can imagine some example hardcoded implementation (that you really shouldn't take too seriously because it's just an example and no real AI code would ever actually look like this):

if (distanceToPlayer < FLEE_RANGE) { 
    moveToCover(FAR)
} else if (inCover() && distaceToPlayer < FIRING_RANGE) { 
    shootAtPlayer()
} else {
    moveToCover(NEAR)
}

As the YAGNI principle suggests, this is probably a good starting point, and can even scale to many different enemy archetypes through appropriate use of inheritance and composition. 

But when you have a huge number of enemies and want to afford greater flexibility to your game designer, more of this logic will need to be driven by data. You can start to imagine how such a system would look. Games like Dragon Age have even exposed such a system to the player.

class Rule {
   Attribute attribute
   float value
   Action action
   Operator operator
    
   boolean evaluate() {
       boolean success = false
       switch (operator) { 
           case LESS: success = mAttributes[attribute] < value 
           case GREATER: success = mAttributes[attribute] > value 
           case EQUAL: success = mAttributes[attribute] == value 
           //.. etc.
       }
       if (success) { 
          performAction(action) 
       }
       return success
   }
}

You would then define some rules to govern behaviour on a given enemy through data in a priority list.

NormalEnemy {
    RULE(ATR_HEALTH, LESS, 0.35, ACTION_RETREAT)
    RULE(ATR_RANGE, LESS, 50, ACTION_RETREAT)
    RULE(ATR_RANGE, LESS, 100, ACTION_ATTACK)
    RULE(ATR_RANGE, LESS, Infinite, ACTION_TAKECOVER)
}

Looks pretty graceful, doesn't it? Until of course you add your boss battle that needs ATR_IS_HOLDING_SPEAR_OF_TRIUMPH, and ACTION_PIERCE_ROCK_OF_DESTINY_WITH_TRIUMPH_SPEAR_FOR_GREAT_JUSTICE_AND_VICTORY_COMMA_MOTHERFUCKER.

All joking aside, a system like this can be very useful and has its place. Deciding to make a system like this could potentially save you large amounts of time. What I recommend as a general rule however - don't start here.  Scale up to it.

Why scale up?

Some games, like Dragon Age, the AI tactics are part of the design and you will likely jump right into making a robust system like the one above.  But when you're unsure, it's a safer bet to start hardcoding and scale up from there if necessary. This has several advantages.
  • You will have a working prototype faster
  • Making the hardcoded system will help inform the design of your robust one
  • You may not need the bigger system
  • It is very difficult to scale down if it turns out you don't need it
  • An overarchitected system can be cumbersome for coders and designers
An important caveat

It may be tempting to use the YAGNI principle as a justification to avoid writing clean code. Your argument might be that it will take so much longer to write my code cleanly and we aren't going to need it, so therefore I shouldn't bother.

Unfortunately this logic is predicated on the false assumption that writing unclean code is somehow faster than just writing it cleanly to begin with.  Don't use YAGNI as an excuse to copy/paste your code or violate other various clean programming practices. Use it to avoid unnecessary features or overarchitected systems.

To summarize

For new features you should be doing a cost benefit analysis wherein you ask yourself:
  • Do I need this right now?
  • And if not, does it cost anything to defer the work until later?
Usually, the answer is no - you should deferring the work. 

And for architected systems, a good rule of thumb is to err on the side of simpler code with the intent to scale up later if necessary.

Optimization

I wanted to touch on one more subject as it is closely related to the YAGNI principle, and that's optimization. You've surely heard the phrase "premature optimization is the root of all evil" and it's absolutely true in many cases. 

(the full quote is: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%" - Donald Knuth, Computer Programming as an art, 1974)

The basic premise is that it is a waste of time optimizing something that may ultimately be deleted, or turns out to be not a huge consumer of performance in the end. Premature optimization also suffers from the other problems I already mentioned of potentially reducing clarity, introducing bugs, or increasing complexity, for little if any benefit.

However, there are real cases where it makes sense to optimize as you're writing, and deciding whether you should be optimizing or not is a big subject that deserves its own article.  That being said it's still related to the YAGNI principle so I wanted to at least mention it here briefly.

Conclusion

That's all folks - this is a pretty broad subject that rears its head frequently, has potentially huge efficiency consequences, and of course offers many angles for further discussion.  Hopefully this introduction will at least familiarize you with the concept and give you some ground rules that can put you in a good position to make effective decisions regarding YAGNI, worst acronym ever.

Tuesday 28 July 2015

Effective Coding: Don't Repeat Yourself

I've written about making sound effectswriting music, and even creating pixel art, all from the perspective of a programmer, so I figured at some point I should actually write about programming.

I was going to write the typical article that reads "8 weird habits of effective programmers, number 5 will literally make your brain explode", but then I realized I have really a lot to say on these subjects and a brief overview just wouldn't be sufficient.  

So, instead I will be doing a deep dive into one particular subject, and if enough people find it useful, I'll write more articles like this.  

But first, if you're one of the many people who learns better by listening and watching, I put together a video that goes over all these points.




So what's the subject? 

I'll be discussing a programming principle called Don't Repeat Yourself, or "DRY". It's an extremely useful principle to follow for all programming, not just game code.

The basic premise is to never have duplicate code in your program. It may sound obvious, why would anyone just copy and paste code? But code duplication is actually a surprisingly common programming problem that I've seen from both beginner and senior level programmers.

This typically manifests itself when you want to write two functions that are similar, but have a few differences. Instead of following the DRY principle, the programmer in question will copy and paste the function, only to change the few lines that are needed.  This leaves much of the code in the two functions duplicated.  

Consider this example of Code duplication:

shoot() { 
    ammo--
    spawnProjectile()
}

shootUnlimited() { 
   spawnProjectile()
}

In this example, spawnProjectile() is duplicated. Yes it's just one line, but it's a contrived example and represents potentially many lines of copied code.

Why is this a problem?

Code duplication is very bad for several reasons.

1. It's a maintenance nightmare

Every time you update your code, you have to then find everywhere else you have copied it and update it there also. This is especially problematic if you are working with a team of programmers who may need to edit your code - how will they know that you've copied it, and where?

The maintenance problem leads into problem #2:

2. It introduces bugs

When you update your code, you might accidentally miss one of the spots where it has been copied, causing an error. You might correctly update all the spots, only to realize that the new code wasn't intended to be copied elsewhere.

Sound confusing? It is. In fact, that's problem #3:

3. It's confusing

It's difficult to parse multiple similar functions in your head. Why are there so many? What's the difference? Which one do I call? It's just more total code that your brain needs to read and process. Less code to do the same thing is often better, and avoiding duplicated code is always better.


Common rebuttals

For much of my career in the games industry I've been a lead programmer, which means I am often reviewing other people's code and potentially asking them to change it... for the greater good!  All too often, programmers will become defensive and try to justify why they have violated the DRY principle. These are some of the common rebuttals excuses I have encountered:

1. It's impossible to write it any other way

Beginner programmers will give you this one. "It can't be done!" they will say. Here's an example which will elict the response: "There is no way to write this without copying that line".

if (isEmpty()) {
    if (canReload()) {
        reload()
        spawnProjectile()
    } else {
        playEmptySound()
    }
} else {
    spawnProjectile()
}

There is always a way to avoid code duplication. Sometimes it's as easy as rearranging the code.

if (isEmpty() && !canReload() {
    playEmptySound()
} else {
    if (isEmpty() && canReload()) {
        reload()
    }
    spawnProjectile()
}

The programmer might protest further... "Ah but now you have a condition copied! You wrote isEmpty() && canReload() twice! Sort of! See? It is impossible!"

So you write...

if (isEmpty()) {
    if (canReload()) {
        reload()
    } else {
        playEmptySound()
    }
}

if (!isEmpty()) {
    spawnProjectile()
}

More protests might be coming... "But that's not EXACTLY the same"... etc. but I think the point has been made; there are many ways of writing the same block of code, and you want to choose the one that has no code duplication.

As much protesting as the above excuse will give you, this next one is far more difficult to deal with:

2. It's faster "for me" if I write it this way

I'll go on a quick tangent here to say that I have a large number of programming principles like DRY that I follow and expect other programmers working on my projects to follow. The people who follow them are not just producing fewer bugs, they're also faster. Sometimes orders of magnitude faster than their copy & pasting colleagues.

Here's why it is not faster for you to copy and paste your code:

1. You are likely not considering how much time you spend in the future updating all the spots you have copied your code

2. You are also probably not considering all the time you will have wasted (your time and other people's time) fixing bugs you have introduced

3. It really just isn't faster to copy & paste and find the spots you want to change and re-write them than it is to just write the whole thing without code duplication to begin with.

If you're finding that it takes you a long time to write code without duplication, you just need more experience. Do it more, practice it more. Soon it will be like second nature and much easier. Ironically, my guess is you will probably find that you are faster too.

Experience is an important point here - some people have been coding for a long time, but are used to the way they code and aren't really improving. Programmers in this category are often the culprits for this next excuse:

3. It's okay if I only do it recreationally occasionally

Sometimes programmers are experienced enough to know not to duplicate code, but then do it anyways. The reasons given are always the same: "It was only 5 lines" or "I only do this once in a while" or "ARE YOU JUDGING ME? I CAN QUIT WHENEVER I WANT!" Well, maybe not that last one.

Truthfully, I've rarely heard a reasonable excuse for copying and pasting. I've even seen the trusted "I only did this because Cert is tomorrow!" fail before because, prophetically, the code in question was broken and needed to be patched.

The whole idea is to get into the habit of writing robust code so that you are comfortable with it. So if you find yourself feeling the urge to just "quickly copy and paste" something, that's a great opportunity for you to stop, reconsider, and improve your abilities.

Solutions

So we know it's bad, we've debunked some common rebuttals, so now how do we avoid code duplication? There are different approaches depending on the type of code you are working with.

1. Rearrange

As seen in the above example, sometimes it is possible to just rearrange your code to avoid a copy and paste. In particular, if the code in question has no order dependency, it may be possible to group all the duplicated code into a similar block, then split it into a function.

Consider these two heinous functions:

shootOctopusGun() {
    ammo--
    if (isEmpty()) {
        reload()
    }


    playSquishySound()
    playInkFX()
}

shootMonkeyGun() {
    ammo--
    if (isEmpty()) {
        reload()
    }


    playHollerSound()
    playBananaFX()
}


Since the common code is all adjacent, you could very easily split it into a separate function.

shootCommon() {
    ammo--
    if (isEmpty()) {
        reload()
    }
}

shootOctopusGun() {
    shootCommon()
    playSquishySound()
    playInkFX()
}

shootMonkeyGun() {
    shootCommon()
    playHollerSound()
    playBananaFX()
}


Of course, you still have duplicated code here - shootCommon(). It's not really removing the code duplication, but rather shrinking it.  Also, making too many functions like this can also be confusing. So in the example shown, this isn't even close to the best solution. Don't do this. We'll slowly improve this code until it looks quite nice by the time we're done.

There are of course cases where using functions makes sense. Functions are useful when you have common code that must be strewn about many different areas. For example, a Vector::getDistance() function. I think most programmers understand the typical use case for functions like this.

2. Conditions

Another alternative is to use conditions. This is just an if-else or switch-case block that flips on some condition. It's a slightly better way of writing the above example as the code duplication finally disappears:

shoot() {
    ammo--
    if (isEmpty()) {
        reload()
    }

    switch (gunType) {
        case OCTOPUS: playOctopusFX(); break
        case MONKEY: playMonkeyFX(); break
    }
}

playOctopusFX() {
    playSquishySound()
    playInkFX()
}

playMonkeyFX() {
    playHollerSound()
    playBananaFX()
}


Marginally better, but not great. We'll continue to improve this later.

Conditions are useful when you want some change to a (usually small) part of your function, like the following:

drawString(String text, Align alignment, Coord position) {
  int width = calcWidth(text)
  int x = position.x
  switch (alignment) {
  case CENTER: x -= (w / 2); break
  case RIGHT: x -= w; break
  }

  // Drawing code below...
}

In this case, the calling code gets to decide how the function behaves, which is useful. You may want to draw a string from many different alignments, and the onus is on the caller to decide what alignment is appropriate.  The bulk of the drawing code remains the same, and simply uses a new x-coordinate based on the alignment property.

I don't recommend conditions when the content of the condition is drastically changing the behaviour or meaning of the function. In those cases, you may want to use:


3. Inheritance or Composition

By now, OOP (Object-oriented-programming) junkies are chomping at the bit screaming "USE INHERITANCE!111 A FLJSLDKFJ".  There are some situations where inheritance or composition is a great solution.  For now, we'll skip over composition, as deciding when to use composition over inheritance merits its own entire article.  

So how does our still-ugly code example look with inheritance?

Gun::shoot() {
  ammo--
  if (isEmpty()) {
     reload()
  }
}

// OctopusGun and MonkeyGun derive from the Gun class
OctopusGun::shoot() { 
  Gun::shoot()
  playSquishySound()
  playInkFX()
}

MonkeyGun::shoot() { 
  Gun::shoot()
  playHollerSound()
  playBananaFX()
}

Better still. Code duplication is still gone, and we've removed that unnecessary switch-case statement. Furthermore, we've divided the custom FX code into separate classes, which can be a nice way to keep them organized.

But, as I'm sure you can guess, we can do better. More on that next.

Using inheritance (or composition) is often very clean, but I don't recommend it in the above example. Now I need to add new code and a new class for every different type of gun, when the only difference might be the kind of FX it creates. So how can we improve this?

4. Use data

Rather than write new code and new classes to define how each gun might differ, it is often possible to just use data to drive how your code behaves. Let's see how this looks:

// in Gun class
int ammo

// Serialized denotes that these variables are read from data
Serialized  {
  int numAmmoConsumed
  Sound firingSound
  Effect firingEffect
}

shoot() { 
  ammo -= numAmmoConsumed
  if (isEmpty()) {
     reload()
  }

  playSound(firingSound)
  playEffect(firingEffect)
}

// in WeaponData.dat
[Gun]
numAmmoConsumed=1

[OctopusGun]
firingSound=Squishy
firingEffect=Ink

[MonkeyGun]
firingSound=Holler
firingEffect=Banana

[EnergyGun]
numAmmoConsumed=0
firingSound=Zap

firingEffect=MuzzleFlash

Now we're talking. The shoot() function is short, has no inheritance, only 1 condition, is straight-forward and easy to debug.  Perhaps a game designer even came in and added a new weapon - EnergyGun - and they didn't even need source code or any programming knowledge. In fact, they were able to add that weapon in real-time without needing to recompile or even restart the engine. That's awesome!

Of course, you can have too much of a good thing. Using exclusively data to define the behaviour of 100 unique guns isn't going to be clean. You're going to end up with a single enormous Gun class filled with code that might pertain to only 1 of those 100 guns. 

So when do I use what approach?

Some rules of thumb(s):

1. Use Data and Base-class code to drive functionality that is common to most types

Example: playSound(firingEffect)

2. Use conditions to make small functionality changes throughout your code

Example: if(isEmpty()) reload() 

3. Use a derived-class or composition to implement functionality for a group of similar types whose behaviour differs sufficiently from the norm.

Example (this time we'll use composition): 

shoot() { 
  firingType->executeFire()
}

// Two examples of firing code that are different enough in behaviour such that they each warrant a new class
ProjectileFiringType::executeFire() { 
  Projectile p = spawnProjectile()
  p.setVelocity() // .. etc.
}

TraceFiringType::executeFire() { 
  Result result = physics->rayTrace()
  if (result.hitEntity) // .. etc.
}

Conclusion

Hopefully this post helped shed some light to a very useful programming principle - Don't Repeat Yourself. I have a huge list of programming principles like this one, so if enough people find this useful, I will write more articles like this :)

Share your thoughts by commenting below!

Saturday 4 October 2014

8 tips for a smooth launch to your indie game

It's time to finally launch your game... or is it? Have you done everything on this list?

In the game industry there is an oft-quoted rule of thumb that states "Creating 80% of the game only takes 20% of the time". It's a variation on the Pareto principle, and in my experience, it is fairly accurate.  Even once your game is finished, fully playable, and not to mention, fun, there is still a ton of work to do before you're ready to hit the trigger and let the rest of the world play it.

I've been a key developer on the launch of several major games, even more small ones, and of course the only developer for Tactic's first (and currently, only) game, Immortal Empire.  I've seen smooth launches, bumpy launches, and delicious lunches.  Let me make this next point abundantly clear by writing it in the largest, boldest font this website will allow:

Having a smooth launch for your indie game is extremely important.

Your dev-instincts have probably already alerted you to this, but let me just drive it home with a few important factoids and common misunderstandings.

  • This rule does not really apply to major studios. If you want to point to Skyrim being buggy, or World of Warcraft's bumpy launch despite their respective successes, please understand that this doesn't apply to you. This is because:
  • People will not wait for your game to work. You will get a big boost to traffic when your game launches. If your game is not working for a particular user, that user will very likely never ever play your game again. Boost to traffic wasted. And it's not just the users:
  • Distribution platforms measure you based on a very brief period during your initial launch. Platforms like Steam or Kongregate will decide how much advertising your game will subsequently receive based on how it performs at launch. At launch. Not 2 weeks after launch, and certainly not 6 months later even if you significantly update or improve your game.  And finally:
  • Game reviews will come at game launch. Even getting someone to review your game can be difficult. Getting them to re-review it after you've fixed all the bugs? Forget it. And expect a bad review.

If the above doesn't make it clear - don't don't don't launch a half-finished or even 90% finished product with the idea that you will "continually update it" as it grows into a massive success. Finish your game, then launch it. If it's a success, update it and grow your community.

One key clarification. When I say launch - I'm referring to your "big launch".  For example, when you go live on Steam to the general public. Closed betas, alpha funding, and other soft launch techniques are actually something I'm about to recommend.

So you're launching a massively multiplayer game and don't exactly have thousands of trained quality assurance testers to make sure it's working great. How do you make sure you have a smooth launch?  It's a mix of prevention and damage control.

Tip 1 - Have a fast build / deployment system  

Your bug-free game has lots of bugs. Sorry :(.  Don't believe me? The most recent patch for Starcraft 1 was released in 2009, over 10 years after its original release date. Fixing a typo? Nope - just little things like being able to nuke from anywhere on the map.

Sometimes your patch might even introduce bugs!

You're going to need to patch your game, so make sure that you have a fast, 1-touch build and deployment system ready so you can catch anything that slips through the cracks immediately.

I've seen people try to follow a complex 20-step process just to build and package up their game for deployment, and the whole time they are quadruple checking each step because they don't want to make a mistake.  Don't do this! Automate your build and deployment process. Write a program, a script, anything! Throw some (preferably lots of) automated testing in there too! It is much faster, safer, and gives you confidence that your game has been deployed properly.

Most importantly, it means you can very rapidly fix bugs as soon as they are discovered.  Which brings me to:


Tip 2 - Have an automated bug reporting system  

So we've established that your game will have bugs, despite your best efforts.  The vast majority of people will never tell you that they have encountered a bug. During the launch window, if it's a severe bug, or your server has crashed, they will most likely just silently quit your game and never play it again.

Use asserts, detect crashes, keep a useful running log of user actions on the client machine to aid debugging and upload reports to your web server.


Tip 3 - Implement real-time memory and performance profiling

Some bugs are simple and easy to reproduce.  Others are more insidious.  Your game might be slowly leaking memory or fragmenting. Your performance might be worsening over time or spiking in certain situations you hadn't previously encountered.  Not only is it useful to have in-engine real-time profiling tools for development reasons, it is very useful for catching these sorts of bugs and fixing them.

Keep track of poor performance or unusually high memory and report such cases using your automated bug reporting system.

It's also important to check these things during development. So you should:

Tip 4 - Run automated tests during development

"yo jsut hire a bonch of roobots to test ur game11`" - xXgamedevkilla16Xx
I know you're eager to build weapons and enemies, but you should spend a bit of time building some robot friends - an automated testing system. Do this early in development and it will save you mountains of headaches - and dissatisfied customers - in the future. 

You should be running the 3 main "S"'s of testing: Smoke, Soak, and Stress tests.

  • The type of smoke tests that I recommend are basically a barrage of unit tests. Check to see if everything works at a basic level. 
    • Everything from "Does the game run?" to "When I execute a trade with another player, do they end up with the expected item?"
    • Usually a programmer will manually test the code they just wrote for its direct functionality. Smoke tests are excellent for catching situations where the code you wrote has inadvertently broken other areas of code. 
    • (PS - proper code architecture can minimize this, but that's for a future blog post.)

  • Soak tests will generally run the game for extended periods of time looking for memory bloats, leaks, performance leaks and such.
    • Boot the game, and leave it running executing simple "normal" actions for long periods of time (say, 24 hours. Weeks if it is a game server.) For example, log a player in. Say some things in chat. Enter a game room. Kill a monster. Quit. Repeat!
    • Boot the game and load all your levels in a loop over and over again. This might reveal leaks caused by level transitions.
    • Boot a level and warp to various areas in the game, (in particular for 3d games) looking for performance leaks. Boot next level, and repeat.
    • (PS - even garbage collected languages or engines can experience memory bloats. You're not safe! Run soak tests!)

  • Stress tests attempt to maximize a particular element of your game. This will give you a good read on the capacity of your game and reveal problems that otherwise would remain hidden until an end user experiences it.
    • Fill a level with highly intensive graphics to stress the GPU. Use realistic situations - the goal isn't just to overheat your video card. Spawn more enemies, have them all firing weapons and blasting particle systems, and so forth. 
    • Fill a level with highly intensive computations to stress the CPU. Again, use real-world situations to test your game, we're not here to just eat up cpu cycles. Run complex physics simulations with thousands of entities. Execute huge numbers of costly pathing lookups for units in the game.
    • Fire large numbers of database accesses, or login large numbers of users to your multiplayer game, flood bandwidth with large amounts of network traffic (or, better yet, limit your bandwidth to see how your game performs under poor network conditions).

You might think the cases you're developing will never happen and aren't worth testing. This is exactly the fallacy that causes bugs to slip through the cracks.  

Run these tests regularly. Daily, or with every build. Some companies will run this on every changelist that's submitted. Running tests frequently lets you isolate what changes you made that may have contributed to degradation of memory, performance, and most important, stability.

These tests also help identify bottlenecks. At one point when we were developing Bioshock 2 multiplayer, if every player equipped the incinerate plasmid and blasted each other, performance would grind to a halt. We found this case using a stress test and optimized that plasmid until it performed much better. 

When I first ran stress tests on Immortal Empire I saw very quickly how many concurrent users the server could potentially handle, and found that database accesses and unit pathing were bottlenecks. So, I optimized all the DB accesses and offloaded those calculations to a separate server. I also severely optimized the pathing to handle common failure cases (where no path was found) as they were the most costly operations.


Tip 5 - Soft launch your game. Repeatedly.

Houston, we have a pillow.

Other than QA and automated testing, your best defense against a bad "hard launch" is to soft launch your game as much as possible.  The idea is simple: launch your game to a select group of people who are aware your game is a work in progress.  People are going to do things your automated tests aren't.

Even with all the automated testing, Immortal Empire did the following and found bugs at every step of the way.
  • Closed beta for friends and family (~50 people)
  • Closed beta for select users (~300 people)
  • Open beta with small press release (~2000 people)
  • Soft launch Kongregate (~10000 people)
  • "Hard launch" Kongregate (~100000 people)
  • Steam (Still to come!)
In particular, you will begin to discover exploits.  Have a plan in mind for what you will do once someone figures out how to dupe items, farm huge amounts of gold in a short time frame, or hack your game to gain an edge over an opponent. Because it's going to happen!

Friends, family and select users will be vastly more forgiving than the average public. For us, the "open beta" and "soft launch Kongregate" included random users and let us gather game statistics too - what immortals were most popular? The most powerful? What spells seem to be dealing too much damage or too little? Which items are people choosing? Are they too strong? This type of data mining helps tremendously to balance your game and make it more fun and fair for everyone.

The remaining tips pertain only to multiplayer games. I have released many many real-time multiplayer games in my career and can confidently say they have their own set of unique, much more difficult problems than single player games.

Tip 6 - Schedule maintenance with a shutdown warning for connected users

Don't just tear down your server if you've found an exploit or crash.  Schedule maintenance, allow it to continue, and start fixing the problem.  When the maintenance window comes around, all you'll have to do is use your fast deployment system and boom! You're back in action.

Players don't like their game to be down for extended periods of time, but if there has to be a shutdown, they are far more forgiving when it's scheduled in advance. This gives them confidence that the shutdown was planned and that everything will be running again soon.

When it is time to shutdown your server, don't just disconnect players in the middle of a game. Give ample warning that the server is about to go down, and disable creation of new games. Wait until most of the games are finished and then tear it all down.

Tip 7 - Use separate staging and release game servers

This is the most optional tip in this list, but it's pretty handy as it will allow you to patch your game server with no perceptible downtime to your end user. 
For you programmery types this is double buffering for game servers.

Basically you have 2 sets of servers running concurrently, one for staging and one for release.  No one is logged onto your staging server, only the release server. It's time to patch! Update your staging server and leave your release server running. Take as much time as you need, because no one is logged on. Once it's ready to go, redirect all new traffic to the staging server.  

Then, as players finish up their games and log off the release server, the total number of users will eventually reach zero.  Now, switch them! Your staging server becomes the release, and the old release becomes staging. Voila! You just patched your game code and no one had to endure any downtime.

Tip 8 - Put a hard limit on the total number of concurrent users


Believe it or not, people actually will happily wait in line for something. But if you let them in and realize it's too full and ask them to leave, that's not going to go over well.

It's true that you want to have the capacity to support unlimited number of people. But you can't reasonably expect to estimate how popular your game will become, and server costs can be expensive, so overcompensating is unwise.  It's better to tell someone "sorry wait in line" than have them overwhelm the server causing it to screech to a halt.

Use your stress tests to get a sense of your server capacity for network bandwidth, CPU consumption, and memory. Then, cap the number of users to ~80% of that limit.  Finally, use your real-time memory and performance profiling to measure live users to get a sense of whether your estimated capacity is accurate.  Code in support to adjust that capacity real-time as needed.

Traffic for a new game is typically very spiky. On Immortal Empire we were running a small open beta and averaging ~20 concurrent users. Overnight, we had a favourable news article appear on Destructoid. I woke up to ~200 concurrent users and a slew of bugs.

Fortunately, I had prior experience launching multiplayer games and was ready! I followed the above tips and the server kept running despite the unexpected spike in traffic during our soft launch.  Then I scheduled some maintenance and released a patch a few days later.


Conclusion


So the million dollar question: Is it worth it?  I'll be the first to admit that if you're writing a small, single player Tetris-clone, you probably don't need to implement anything on this list. I've released many small scale games that didn't have these systems in place and they launched fine.

But as the complexity of your game increases, the above tips rapidly become requirements.  I've been to many major game studios, and these systems are commonplace. For any medium scale or greater game, or any multiplayer game with centralized servers, you're going to want this stuff.

You put so much effort into your game, you'd be throwing all that effort away if you aren't properly prepared.  

Thanks for reading, and I hope this helps you have a smooooooooooooth peanut butter lunch!  I mean launch.

Addendum

Immortal Empire was greenlit on Steam back in July 2014 and we're gearing up for another "hard launch". We're currently running a Kickstarter to help fund some significant improvements to the game before we go live on Steam. A stretch goal on that Kickstarter is to release our game engine and source code for free. 

So if you'd like to see how we implemented the above features, or simply want to use our game engine (for free!) to make your own game (and never have to build any of the above systems) then mosey on down to our Kickstarter and give it a look-see. Thanks!

Tuesday 24 September 2013

How to get art in your game without being a professional

I'm not an artist, but I need art in my game.

This is part 3 in a series of articles describing how I approached each aspect of creating Immortal Empire as an independent developer. Read the whole backstory here.

Artwork is the area I relied most heavily on outsourcing to a number of individuals, most notably Mat Chambers (characters), David Baumgart (spell icons, fx), Zenobia Homan (tiles, environment objects), and Eric Vedder (illustrated characters). A few other artists (Sergei Churbanov, Cecilia Santos, Jasmin McGahan, David Scott) helped out as well to which I am eternally grateful.

Of course, trying to coordinate so many different people was difficult, and in some cases I needed to fill in some gaps with artwork myself.  In my previous article where I discussed the music to Immortal Empire, I pointed out that I have some formal experience with music, making that aspect of the game much easier to tackle.  When it comes to artwork however, I have literally no experience to draw upon. (moar puns!!1)

In the end, I deigned to create some affectionately termed coder art, much of which can be seen in the final product.  This article describes the process I went through and includes some tips I picked up which are hopefully useful to other artistically barren programmers like myself who need art in their indie games.

I'm very proud of many things in Immortal Empire, but let's face it, the artwork is not exactly cutting edge. At a fixed 800x600 resolution and consisting almost entirely of hand-drawn 2D pixel art, Immortal Empire's artwork lands somewhere between X-Com and Warcraft II, meaning it would feel right at home if it were released in 1995.  Alas, it was released in 2013.

Click for full-size
Many games have managed to be very successful in recent years despite not having advanced graphics, often employing a minimalist style to great effect.  Unfortunately I feel as though minimalism just wouldn't have worked for the game design of Immortal Empire, so instead I attempted to mimic a style that is very nostalgic for me: early 90s PC games.

Onto the artwork. I contributed in a number of areas, but I'm going to focus on a couple images I drew that will best describe the process.  Here is the animation for the entangle spell on the left, and the idle pose for the Elder character on the right.


For the pixel art, the process I eventually settled on went something like this.

0. Compare against a couple different neutral colour backgrounds, never change your opacity, and always work on a separate layer.  Accidentally using alpha or manually blending into a fixed colour background looks terrible in game, and it is a very easy mistake to make. Using separate layers for each step lets you restart when you make mistakes and do A/B comparisons easily.


1. Start with an outline.  When drawing lines (without alpha transparency), I found it almost always looked better to leave diagonals open rather than filled in like the image on the right. 


When creating rounded lines, decrease or increase the number of straight-line pixels consistently, rather than doing something arbitrarily. I'll often draw the line free-hand at first, delete the diagonals, then touch up the curve to keep line lengths consistent.


This might sound obvious but you really don't want to move onto the next step until you have finished this one.  At this stage, a silhouette can work as well, but I prefer an outline so you don't lose interior details.

2. Add shading to your outline. As with the lines widths, be consistent in how you gradient the colour.  By that I mean gradually increase or decrease the luminosity rather than flipping back and forth or skipping a shade.  I try to avoid adding detail in this step, instead I just shade uniformly. When creating rounded things like the tentacles below, I found it often looked best to put the brightest section near the center, not right on the edge. Keep a palette of distinct colours you have used on a separate layer for future use.


3. Add detail to the shading. This is where you can put creases in clothing, make things appear rough or shiny, or add dithering if that is your art style.   I try to avoid introducing new colours here, and instead would refer to the palette that I created in step #2.

Click for full-size
4. Add final touches.  Here I add little doodads like the buckle on the Elder's sash or the thorns on the tentacles. When doing this step, I found I often had to go back and adjust the shading.  For example, creating a little shadow underneath the buckle made it feel less flat.  Only when I am 100% happy with the look of this one frame will I start another frame of animation or attempt to draw more of the same element (such as the multiple tentacles in the entangle spell).

That's it!  My preferred setup is to use two windows, one zoomed in very close where I edit the pixels, and another zoomed out to 100% so I can actually look at the results. I check the actual zoom distance frequently, as I found that things sometimes look great zoomed in, but from afar look totally wrong. In fact, the opposite can also be true!

Like most things, art of course takes years of practice to improve. But here's hoping these few tips can help other artistically challenged independent developers like myself create some medicore but passable artwork for their games.  If you have anything to add about your adventures in this regard, or tips you can suggest, please comment below!