RSS Feed IDS on YouTube IDS on Facebook IDS on YouTube

The Importance of Performance Testing

posted on November 21, 2013 at 9:30 pm by Steve Karolewics

Imagine for a moment that you’ve just spent a year creating a game, fixed hundreds of bugs, and you finally release it. You’re worried about a lot of things: Will anyone play the game? Will they like it? Will there be any problems? As it turns out, the answer to that is usually yes to all three to some extent. But there will always be problems after launch.

For Minecart Madness, our launch issue was pretty worrisome: on first generation Windows Phone 7 devices, the game ran slowly. Very slowly. So slow that it was unplayable. So what happened?

When Windows Phone first launched in 2010 with such phones as the Samsung Focus, they were more than capable of running games at 30fps, but compared to the next generation (like the Nokia Lumia 800), the original models were slow. Approaching the launch of Minecart Madness, we did most of our testing on our day-to-day phones, which were not first generation Windows Phones.

Looking for Solutions

We immediately began searching for ways to speed up the game on those slower phones. The first thing to go involved the minecart’s headlamp – we use a second rendering pass to draw it with additive blending. Even after disabling that, at certain times we still saw slowdowns, so we chose to reduce the rendering size from the then-native 800×480 pixels.


In particular, whenever lots of track supports were on screen, we still saw slowdowns even with the headlamp blending disabled.

An across the board cut on pixels sounds drastic (and it is), but other possible solutions involved more re-architecting than we wanted to do. After all, every day that the problem persisted were potential lost players and negative reviews. With a bit of empirical testing, we determined that 60% size would allow the game to run fast enough and still look acceptable. Luckily, with XNA we were able to scale all rendering with a single parameter change.


At 60% size, there’s a noticeable quality drop (though less so while playing), but everything is still readable.

So now we knew how to fix the problem, but we needed to figure out when to fix it. See, we wanted to fix the problem for older devices, while still providing the pretty visuals for newer phones.


The most obvious solution was a blacklist of phones, and explicitly checking for the first generation devices. The downsides to this approach should be obvious: it’s easy to be too aggressive or not aggressive enough with the blacklist, and it comes with a maintenance cost of keeping it up to date. We turned away from this plan almost immediately.

Framerate Checking

Our next attempt focused around the fact that the first few seconds of every Minecart Madness run are the same – a flat track with 3 coins and a shield power-up. If the game doesn’t run at 30fps for the first second or two, then there’s no problem. It’s only when the player is trying to jump, dash, or navigating the randomly-generated track that it becomes painful to play.

By measuring the time between beginning to draw and ending (for XNA, SpriteBatch.Begin and SpriteBatch.End), we planned to check if a frame was taking “too long”. At 30fps, each frame should be 33.33ms, so we gave ourselves some breathing room and marked a frame as “slow” if it took longer than 37ms. Then, if we have more than 10 slow frames in a given time interval, we switch to the “low quality” mode outlined earlier in this article.

We thought that with this approach, we would be ready to publish the much-needed performance update. However, even when running on a first-generation Windows Phone, we were not falling back to low quality. Well, why not?


In most modern rendering APIs, a process called batching greatly improves performance. Rather than sending each drawing command to the GPU individually, the API will collect many calls together, until either rendering is ended, or the queue of command data surpasses an internal limit.

You might think that for XNA, SpriteBatch.End would be the call where rendering would actually occur. The documentation seems to agree with that reasoning. However, as it turned out, we were missing a lot of the processing time. To fully capture the time for a frame, we begin measuring at the beginning of Draw, and added an override for Game.EndDraw. By finishing our measurement here, we ensure that all rendering time is included. After making this change, the first-generation phones drop to low quality very quickly, providing a playable experience.

Handling Hiccups

We noticed that even on the newest Windows Phones, sometimes we would still fallback to low quality. Sometimes this was due to background processes on the phone, and sometimes it was due to the amount of rendering we were doing, such as the screenshot above when there are lots of track supports to draw.

To account for this, we reset back to high quality each time the player begins a new round. Then, if a fast phone just happens to have an unfortunate set of frames, next game it will switch back to looking crisp. Slower phones also reset to high quality, but as expected, they jump to low quality quickly for each round.

At this point, we finally had a runtime solution for detecting performance problems, and correcting on the fly to provide a great gaming experience for all of our players. We were able to push the update out only a week after the game launched, an impressive turnaround time in our opinion for such a complicated issue. Since we published this update, we haven’t had to touch this code and the complaints about the game being unplayable have completely dropped off.


As with any solution to a problem, there are wins and losses.  A solution is only worth shipping if the benefits outweigh the drawbacks, and there are always drawbacks.  The might be development time, test time, potential destabilization, or performance costs.  Our solution, while elegant in terms of total code change, is not perfect.  In Handling Hiccups, we talked about how we reset to high quality each round.

On the slower phones, the change to a low quality mode will trigger after 10 frames (1/3 second).  This initial slowness does not go undetected by all users.  When play-testing the fix on slow phones, players noticed that the game, right upon triggering a new run, will stutter.  Users almost unanimously agreed that this was a drastic improvement and a tolerable stutter on their phones.  Users became used to the stutter for at the beginning and adjusted accordingly.

Very few noticed the graphical quality reduction, interestingly.  We did.  A fast motion game, such as Minecart Madness, we have learned, has a benefit of masking low graphical quality.  We had discussions about lowering the quality of only specific parts of the game, such as the track, and maintaining high resolution for other graphics.  We concluded that change was not necessary.


We made a mistake.  This is a perfect example of development not being over after shipping.  Ultimately, it is not possible to test on every config.  Identifying your potential weaknesses and testing them early and often is crucial.  For Minecart Madness, gameplay enjoyment is heavily dependent on frame-by-frame performance.  We should have been looking for low performance devices.  Is your game a turn-based strategy game such as Civilization?  Testing on devices with different resolutions and DPI such that users can read all text and easily click on all buttons should be regular on the development cycle.

Moving forward, we know we will make mistakes.  We will likely miss bugs and have to send out day zero patches.  However, we hope that what we have learned from this simple oversight and our future projects will never ship preventing users from playing or enjoying our titles.  For our developer friends reading this, we understand that shipping a title and putting yourself out there requires confidence and resolution.  However, never be so confident in your work that you overlook simple bugs.  Testing may not be the most enjoyable activity when you could be playing your own game or adding new features, but it is crucial to do, end-to-end.


No comments!
Add Comment Register

Leave a Reply

Your email address will not be published. Required fields are marked *

1 + = nine

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>