RSS Feed IDS on YouTube IDS on Facebook IDS on YouTube

→ Category Archives: Minecart Madness

Every developer seems to have that bug. You know, THAT bug. The elusive one that only seems to creep up during playtesting or while testing another feature? This isn’t specific to game development either. Web service developers or even operating system kernel developers have all seen, at least once in their careers, a bug that only seems to pop up when they aren’t looking for it. And it’s not just bad; it’s breaking. Debugging something you can’t figure out how to reproduce is an exercise in balancing sanity with sobriety.

With Minecart Madness, we had the Super Jump bug. We coined that name because the player would jump incredibly high for seemingly no reason. And it didn’t always happen. And we couldn’t figure out why. But it came up when we were debugging issues with track placement. It came up with we were debugging issues with obstacles on the track. It came up when we were figuring out how to handle tunneling. It kept happening. Why?

Filing a Good Bug

Super Jump Bug filing into IDS bug database

Super Jump Bug filing into our bug database

By profession, I’m a tester. My job is to break things and tell you why it’s broken. A really good bug filing should contain the following information:

  • First version where the bug was reproducible
  • Last version where the bug was not reproducible
  • Clear, concise reproduction steps with the minimum steps needed to cause bug
  • Any special machine settings (resolution dependence, high/low memory situation, specific phone, etc.)
  • Callstack if it’s a crash, but can also be useful if it’s not to show exactly where in code the bug is happening
  • Severity, i.e. how breaking of a bug it is
  • Extra manifestations, i.e. does this happen in all situations? For example, if your cross-platform Unity game is crashing on the iPhone, does it also crash in the same situation on Android?

With that in mind, let’s look back at this bug filing. Of the seven tenets listed above, this contains exactly zero. Each of these useful chunks of information rely on a single other event to occur: consistent reproduction. We couldn’t figure out how to make it happen. It just seemed to happen sometimes. Hence, the repro steps above.

What is “Breaking?”

As with all bugs, the Super Jump Bug had to go through a triage process where we decide if the bug is breaking enough to warrant fixing. These discussions usually involve expected amount of time to fix, expected degradation of the user experience, frequency of occurrences, and risk of fix. For the Super Jump Bug, we didn’t know the fix and we’d already attempted to reproduce it and fix it unsuccessfully. Through playtests, users would hit this bug as well.  When a user would hit the bug, he/she would notice immediately and ask what happened. We told those users that we were aware of it and not to worry. We were worried. The feeling of having to downplay a bug you know is breaking and can’t solve is crushing. With that in mind, we did not “Won’t Fix” this bug, if only to satisfy ourselves of knowing we solved it.

Procrastination as a Tool

Like most developers, we decided to just let this bug sit. And sit. And sit. Sometimes, as we’re all aware, one of two things can happen to bugs we let sit. First, the bug “magically” fixes itself. You change some specific values for core mechanics like jumping or movement, increase or decrease your net resolution, and suddenly, the bug isn’t there. It most likely is, but it is being masked or mitigated by your change. A bug that users can’t see, for an indie dev, is not a bug. We don’t have time to make things nice. We have to ship. The second thing that can happen while procrastinating is a revelation. A sudden moment of clarity. For a small moment in time, you become Neo. You can see the code and you realize the bug without ever thinking about it. That is how we solved the Super Jump Bug.

What is Jumping?

All mechanics in games need to feel like an extension to the players mind. Never has this been more true than in touch based mechanics. We touch the character and we want him to jump as we imagine in our minds, run as fast or slow as we picture, or target exactly where we point. Curiously, our minds don’t exactly imagine reality based physics. Not until high school physics classes do we realize that our assumptions about how things move are often wrong. But, this is a game, so we are expected to break with reality. Indeed, jumping should feel as the user would expect, not as it would be on Earth. Therefore, our jump algorithm doesn’t function as a NBA player leaping on the court does.

We break jumping into two parts. First, the initial hop. This happens with just a single touch and is the minimum jump height. This allows for targeted little jumps to cover the smallest of crevices. After that, for a specific amount of frames, we provide an additional upward velocity until either the user releases his/her finger from the screen or our acceptable amount of frames are exhausted. This provides the aforementioned feeling of the character on screen being an extension of the player’s finger, and therefore, mind. This creates a jumping experience that is fun and satisfying. More importantly, though, it makes the player feel liable for all successes and failures regarding hit and missed jumps.

What was the bug?

The code for the jump looked like this in revision 160:

velocity.Y -= _amountTouches == 1 ? HopHeight : fUpwardJumpAcceleration;

Revision 161, after the bug was fixed, contained the following instead:

velocity.Y = _amountTouches == 1 ? -HopHeight : _velocity.Y - fUpwardJumpAcceleration;

Of course, you see the changes, but do you see the bug? Hint: the bug is not on this line at all. To understand the bug, we need to know that, after landing, we reset _amountTouches to 0 in the Minecart’s update method. By doing this in the subsequent update, the Minecart is effectively running a frame behind on determining if it is in the air or on the ground. It goes to follow that a second HopHeight jump can occur.

Reproducing a Bug After Fixing It

Often, we determine a fix for a bug before the bug can be accurately reproduced. Only after conjuring a fix can we deduce a sequence of valid repro steps. For this bug, based on the fix in the earlier revision, the repro steps were as follows:

  1. Jump like any user would, but do not remove finger
  2. Land, keeping finger on screen
  3. Jump again by holding finger (we wait for touch events, never the absence)
  4. Repeat 3-4 times.

Step 3 is by design. We will never not let a user jump while on the ground. After 3-4 iterations of these steps, a large double jump would occur. Every. Time.

Learning and Moving Forward

The Super Jump Bug was a single bug with a single line fix affecting only certain users who happened to touch the screen for long enough in edge cases. However, it highlights that debugging may not be a scientific process. We can see that, because we are changing variables a full update behind, using the built-in debugger wouldn’t have gotten us closer. We tried. But, you can’t hold the screen AND hit break points in the emulator. In fact, reliance on the given tools was a hindrance to our progress for bug #107, the Super Jump Bug.

Brian Kernighan, one of the inventors of C, once wrote, “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” Intimate familiarity with each and every line of your code is paramount to shipping a quality product. Copy/paste is not a design pattern and will lead to bugs. We can confidently say that if we had borrowed another jump algorithm, this would have never gotten fixed. Code simply. Document thoroughly. Be patient. The product and your users will thank you.

Imagine for a moment that you’ve just spent a year creating a game, fixed hundreds of bugs, and you finally release it. You’re worried about a lot of things: Will anyone play the game? Will they like it? Will there be any problems? As it turns out, the answer to that is usually yes to all three to some extent. But there will always be problems after launch.

For Minecart Madness, our launch issue was pretty worrisome: on first generation Windows Phone 7 devices, the game ran slowly. Very slowly. So slow that it was unplayable. So what happened?

When Windows Phone first launched in 2010 with such phones as the Samsung Focus, they were more than capable of running games at 30fps, but compared to the next generation (like the Nokia Lumia 800), the original models were slow. Approaching the launch of Minecart Madness, we did most of our testing on our day-to-day phones, which were not first generation Windows Phones.

Looking for Solutions

We immediately began searching for ways to speed up the game on those slower phones. The first thing to go involved the minecart’s headlamp – we use a second rendering pass to draw it with additive blending. Even after disabling that, at certain times we still saw slowdowns, so we chose to reduce the rendering size from the then-native 800×480 pixels.

Gameplay

In particular, whenever lots of track supports were on screen, we still saw slowdowns even with the headlamp blending disabled.

An across the board cut on pixels sounds drastic (and it is), but other possible solutions involved more re-architecting than we wanted to do. After all, every day that the problem persisted were potential lost players and negative reviews. With a bit of empirical testing, we determined that 60% size would allow the game to run fast enough and still look acceptable. Luckily, with XNA we were able to scale all rendering with a single parameter change.

low-quality

At 60% size, there’s a noticeable quality drop (though less so while playing), but everything is still readable.

So now we knew how to fix the problem, but we needed to figure out when to fix it. See, we wanted to fix the problem for older devices, while still providing the pretty visuals for newer phones.

Blacklisting

The most obvious solution was a blacklist of phones, and explicitly checking for the first generation devices. The downsides to this approach should be obvious: it’s easy to be too aggressive or not aggressive enough with the blacklist, and it comes with a maintenance cost of keeping it up to date. We turned away from this plan almost immediately.

Framerate Checking

Our next attempt focused around the fact that the first few seconds of every Minecart Madness run are the same – a flat track with 3 coins and a shield power-up. If the game doesn’t run at 30fps for the first second or two, then there’s no problem. It’s only when the player is trying to jump, dash, or navigating the randomly-generated track that it becomes painful to play.

By measuring the time between beginning to draw and ending (for XNA, SpriteBatch.Begin and SpriteBatch.End), we planned to check if a frame was taking “too long”. At 30fps, each frame should be 33.33ms, so we gave ourselves some breathing room and marked a frame as “slow” if it took longer than 37ms. Then, if we have more than 10 slow frames in a given time interval, we switch to the “low quality” mode outlined earlier in this article.

We thought that with this approach, we would be ready to publish the much-needed performance update. However, even when running on a first-generation Windows Phone, we were not falling back to low quality. Well, why not?

SpriteBatch.End

In most modern rendering APIs, a process called batching greatly improves performance. Rather than sending each drawing command to the GPU individually, the API will collect many calls together, until either rendering is ended, or the queue of command data surpasses an internal limit.

You might think that for XNA, SpriteBatch.End would be the call where rendering would actually occur. The documentation seems to agree with that reasoning. However, as it turned out, we were missing a lot of the processing time. To fully capture the time for a frame, we begin measuring at the beginning of Draw, and added an override for Game.EndDraw. By finishing our measurement here, we ensure that all rendering time is included. After making this change, the first-generation phones drop to low quality very quickly, providing a playable experience.

Handling Hiccups

We noticed that even on the newest Windows Phones, sometimes we would still fallback to low quality. Sometimes this was due to background processes on the phone, and sometimes it was due to the amount of rendering we were doing, such as the screenshot above when there are lots of track supports to draw.

To account for this, we reset back to high quality each time the player begins a new round. Then, if a fast phone just happens to have an unfortunate set of frames, next game it will switch back to looking crisp. Slower phones also reset to high quality, but as expected, they jump to low quality quickly for each round.

At this point, we finally had a runtime solution for detecting performance problems, and correcting on the fly to provide a great gaming experience for all of our players. We were able to push the update out only a week after the game launched, an impressive turnaround time in our opinion for such a complicated issue. Since we published this update, we haven’t had to touch this code and the complaints about the game being unplayable have completely dropped off.

Tradeoffs

As with any solution to a problem, there are wins and losses.  A solution is only worth shipping if the benefits outweigh the drawbacks, and there are always drawbacks.  The might be development time, test time, potential destabilization, or performance costs.  Our solution, while elegant in terms of total code change, is not perfect.  In Handling Hiccups, we talked about how we reset to high quality each round.

On the slower phones, the change to a low quality mode will trigger after 10 frames (1/3 second).  This initial slowness does not go undetected by all users.  When play-testing the fix on slow phones, players noticed that the game, right upon triggering a new run, will stutter.  Users almost unanimously agreed that this was a drastic improvement and a tolerable stutter on their phones.  Users became used to the stutter for at the beginning and adjusted accordingly.

Very few noticed the graphical quality reduction, interestingly.  We did.  A fast motion game, such as Minecart Madness, we have learned, has a benefit of masking low graphical quality.  We had discussions about lowering the quality of only specific parts of the game, such as the track, and maintaining high resolution for other graphics.  We concluded that change was not necessary.

Conclusions

We made a mistake.  This is a perfect example of development not being over after shipping.  Ultimately, it is not possible to test on every config.  Identifying your potential weaknesses and testing them early and often is crucial.  For Minecart Madness, gameplay enjoyment is heavily dependent on frame-by-frame performance.  We should have been looking for low performance devices.  Is your game a turn-based strategy game such as Civilization?  Testing on devices with different resolutions and DPI such that users can read all text and easily click on all buttons should be regular on the development cycle.

Moving forward, we know we will make mistakes.  We will likely miss bugs and have to send out day zero patches.  However, we hope that what we have learned from this simple oversight and our future projects will never ship preventing users from playing or enjoying our titles.  For our developer friends reading this, we understand that shipping a title and putting yourself out there requires confidence and resolution.  However, never be so confident in your work that you overlook simple bugs.  Testing may not be the most enjoyable activity when you could be playing your own game or adding new features, but it is crucial to do, end-to-end.

One year ago, we launched Minecart Madness for Windows Phone. While we’ve been busy working on a new project since then, we thought now would be a good time to look back and analyze how Minecart Madness has done.

Minecart Madness

Launch

Minecart Madness launched on October 24th, 2012, and the free version launched a few days later on November 1st. We delayed the free version because we wanted to link the player to the paid version (only once per session), and the store link is not resolved until an app is published. It turns out that linking the player to the paid version was not very useful. After all, why pay $1 for a game that you’re already playing for free? Mobile users are increasingly accepting of ads in free games, and our ads were already not very invasive.

We had issues at launch where the game ran terribly on some devices (we’ll cover the technical side of this in a separate post). While we managed to get the problem fixed with a quick turnaround, some damage was already done – we had a few early negative reviews because of this problem.

Ad Networks

Our stance on in-game ads is pretty simple – display and rotate them as often as possible, but never when it would interrupt or frustrate the player. In Minecart Madness, obstacles can appear from the left, top, and right sides of the screen, and the player needs to see the bottom of the screen to tell where they can land. As a result, our only options were to decrease the vertical height of the game, and add an ad in that space, or not display them while playing. The choice was obvious – we only display ads in menus, or between the player dying and starting again. There’s no doubt that this decreased our ad revenue, but so would angering your players, and we care more about the quality of the game than earning a few extra dollars.

Initially, we only used Microsoft’s PubCenter for ads. Despite a pretty good number of ad impressions, we didn’t make much from ad revenue. We discovered a great library called Ad Rotator that let us easily add additional ad networks. We published an update that added InnerActive, AdDuplex, and MobFox.

After testing the waters with these new ad networks, InnerActive was the clear winner, with a much higher revenue rate, and over 70% of our total ad revenue. We can only speculate how our revenue would change if we focused on InnerActive from the start, but for any future games we create with ads, they’ll definitely be our primary ad network.

inneractive

Downloads

Despite less-than-stellar ad revenue, we made this game because it was a game we wanted to play, and we hoped that others would want to play it too. During development, we decided that we would consider 1,000 downloads to be a minimum bar for success.  We were blown away with our download numbers. As of October 24, 2013, we have 1,691,754 downloads! Initially, our download trend matched what we expected – a large initial surge, followed by decreasing downloads per day. However, on two occasions we saw our downloads surge again. To our delight, our downloads have not dropped below 2,000 per day.

minecart-madness-downloads

The top ten countries that downloaded our game are, in descending order:

  1. Russia
  2. China
  3. Mexico
  4. Brazil
  5. Italy
  1. India
  2. United States
  3. Thailand
  4. France
  5. United Kingdom

Those ten countries combined account for 1.22 million downloads. In total, our game has been downloaded in 173 countries, many of which with only a few hundred downloads. However, every download counts, so it was great that we published our game in nearly all available countries.

Brazil is particularly interesting to see in the #4 spot. When we launched Minecart Madness, Brazil required a unique rating outside of the free ESRB and PEGI we obtained, so we did not publish in Brazil. At some point, Brazil began accepting ESRB or PEGI ratings, and after being informed by a Brazilian fan of Windows Phone, we finally published our game in Brazil in late February 2013. This explains one of the later surges in the download chart above. When downloads are compared by amount of time since publishing in a country, Brazil is nearly on par with Russia, so we heavily encourage other developers to release in Brazil.

Reviews

reviews

As already mentioned, we suffered from some initial negative reviews from the game being unplayable on some phones. However, after fixing that issue, we’ve recovered and are sitting at 4 star ratings in 7 of our top 10 countries. We’ve gotten a lot of feedback that Minecart Madness is a lot of fun, but there’s admittedly not much depth to it.

Before we released, we had a lot of grand designs for features to help with this, but we made (in our opinion) the right call to draw a line somewhere and just publish the game. The incentive for continued play could be higher with features such as leaderboards and daily challenges. We hoped to include some of these features in future updates, but we ultimately determined to shift focus to other projects. We still get numerous reviews regarding how addictive the game is, which we attribute to the procedurally generated levels, a differentiator we are particularly proud of. We will cover our strategy regarding responding to reviews in the Windows Phone ecosystem in a future post.

Conclusions

First of all, no matter how large or small the ecosystem for a platform may be, there will always be players. If you build something that you enjoy yourself, you will not be alone. Testing your game should feel like fun, an exploration of your own creation. Shipping is important and you will have to cut in order to pull it off. The journey of making a game does not end with shipping and, in many cases, it is only beginning. Seeing how your art effects players of different ages and backgrounds is exciting, but also humbling. We have learned from this experience and we continue to try to gain an understanding of what we can do better in the future.

We are increasingly appreciative of our players. There are nearly 1.7 million people who went into the store, looked for a game among thousands of deserving, enjoyable titles, and chose to install and play ours. That number increases by over 2,000 every day. While we are making our next game, people are still installing our current one, and that is something we will never forget. Thank you!

A big thanks to everyone who downloaded Minecart Madness. We’ve hit 1 million downloads on Windows Phone!

We’ve been pretty silent lately, mostly due to real life getting in the way, but we are working on some new stuff. With the news that XNA development has been halted, we’ve been looking into new alternatives. Right now we’re getting acquainted with Unity3D and hopefully we’ll be getting started on a new game soon.

Windows Phone users should be seeing an update for Minecart Madness soon. Details can be found on the changelog.