Buick Encore is a mobile augmented reality game, and we created the serverside part running on a Python stack behind the Great Firewall of China.

Buick Encore is a mobile augmented reality game, created by us, Monterosa and Ogilvy Shanghai; we were responsible for the server-side backend part of the game, while Monterosa developed the iPhone and Android apps and Ogilvy was responsible for the game design and overall project management.

The game idea revolves around a virtual car which is placed somewhere in a big Chinese city. Using their smartphone, the players can see an approximate location of this car, as well as the positions of other players, on a standard map. When they get close enough to the car, its exact location appears, they rotate their phone to portrait mode and an augmented reality view which shows the car blended into the real world appears. They can now “get in” the car and “drive it” for up to three minutes; at any time while they’re “driving” they can “get out” of the car and leave it at their current location. The purpose of the game is to find the car and then hide it; at the end of the gaming day, whoever managed to hide the car for the longest period of time wins.

Hide & Seek presentation video from saschaengel_china works on Vimeo.

User login can be done either with Sina Weibo (a Chinese social media platform) or just a regular e-mail address, and the backend can send push notifications to users using Urban Airship’s excellent service.

On the server side, we had to keep track of a rather large amount of real-time data, since both the car’s position and the position of each individual player are updated quite frequently and retained for a relatively long period of time. Each datapoint is rather small, but there are a lot of them per player and they accumulate quickly. Normally, this kind of application with frequent updates and frequent requests for updates from the clients would be very well suited to hosting on Google App Engine. In this case, however, we chose to not use App Engine, primarily because Google does not have a data center in China, and latencies to anything outside the Great Firewall are terrible. The customer was also worried about the potential for getting the application suddenly blocked. Hence, we rolled our own solution.

After some experiments with various technologies (such as REDIS, twisted.web and gevent), our server side backend ended up being written in Python, using:

Tornado’s ioloop
MongoDB, using the asyncmongo Python API
memcache
nginx as a frontend
Stingray for load balancing
Twitter Bootstrap for the admin interface

Since the customer was extremely worried about latency and scalability, we did quite extensive performance tests at an early stage to get a general idea of how much raw performance we could possibly squeeze out of a single server. Based on this we ended up ditching many of Tornado’s conveniences and wrote our own simplistic request handling framework. For interacting with MongoDB, we chose the asyncmongo module in order to avoid blocking our Tornado threads on I/O.

The entire application is hosted on a set of virtual machines at Datapipe’s Shanghai datacenter. We evaluated several hosting solutions before choosing Datapipe; the customer requirement that all machines must be hosted inside mainland China disqualified many options, and in the end Datapipe’s convenient cloud solution and English-speaking customer service won out.

Since this technology stack (referred to by a certain hosting salesperson we spoke to as “the hipster stack”) was mostly unfamiliar to us before this project, we had a lot of interesting lessons to learn, the most important one being that creating web application backends using a completely callback-driven framework is sort of a pain and requires some careful design and planning in order to avoid a massive mess of spaghetti function calls. Most of us haven’t done much serious functional programming since the university courses in it, so this provided a handy refresher course involving things like partials, closures and other such fun.

The Buick encore admin interface

Working with MongoDB in particular has been an interesting experience. It’s certainly very fast and ended up working quite well for our purposes, but we don’t think we’d choose it again for the next project, since it’s simply too tedious to work with. It’s certainly not all bad, but there are a number of small grating issues that makes it feel very rough to handle, at least to someone used to the NDB API of Google’s High Replication Datastore. A lot of this is related to the API we used to interact with the database: I feel a callback-driven async model is much harder to work with than the futures-based model the NDB API uses, but this also has to do with the fact that on App Engine it’s not a problem to block a thread for a few hundred milliseconds since other threads can use the same instance while you’re waiting. However, MongoDB itself also has quite a few minor irritations, particularly related to the generally hairy query syntax.

Tornado on the other hand has generally been pretty smooth to work with. It’s quite easy to roll your own request handling stuff based on its powerful ioloop functionality, but even if you don’t want to do that, the request handlers it ships with are well designed and polished. Our old friends nginx and memcache are nothing new, of course; there’s a reason we keep using them everywhere.

More about the project: Monterosa’s case page Monterosa press release