Cherry Pop: Development Diary

Posted January, 2012 - Andrew Armstrong

About Cherry Pop

Spice up the relationship with your partner by answering exciting and provoking questions you may have wanted to ask, without the fear of being embarrassed.

Cherry Pop helps you discover new activities to do together by asking each person a series of questions in private about their desires, likes and dislikes.

Activities you would both enjoy are then shown to you.
 
Cherry Pop design
 

If either partner declines an activity, it will not be shown, protecting the privacy of the person who liked the activity.

The Android version of Cherry Pop contains sexually explicit questions.

Table of Contents

Idea to Product Phase

The past 12 months have had me writing down every little idea that pops into my head. Trivial or not, I find this habit has helped me spot more things to work on than if I hadn't written them down.

After implementing a few other side projects with no real success, I decided to team up with my friend to work on the Cherry Pop idea I had been sitting on for a while.

In the last four to five months, the project has moved from an idea into a product you can see in front of you.

Game flow design document

Establishing a company, buying a MacBook Pro and learning Objective-C development, writing the API and backend website, teaming up with a designer to develop a presentable product, provisioning servers and iterating our product via feedback has all lead up to this product launch.

A shout-out to the HackerNews community for hosting an awesome environment related to start-ups and industry news.

Company Details

A registered company was formed in Australia (Scramble Media Pty Ltd) in order to lay the business foundation for this project and to allow myself and my friend to work together on projects easier in the future.

Company registration was simple from the point that we used a company registration company (!) to take care of the paperwork.

An accountant's office was used as the mailing address and telephone number for our business, as one is required by Australian law.

Design Philosophy

A designer friend was contracted to worked with us to create a product that looked visually appealing and simple, while steering well clear of very dark reds and other themes commonly associated with (not too great looking) sex related applications in the market place.

We thought one of the selling points of our product should be that it is visually appealing and comfortable to leave installed on your phone.

Development Design

During development the interface on the iPhone, prior to any design work, is fleshed out to get an idea of how the game is progressing.

 
Design during development (programmer art!)
 
Professional Design

With the help of our designer, we were able to come up with several core values that would steer the look and feel of the final product.

We went through several iterations with our designer who was able to come up with several concepts based on our discussions together.

 
Initial design iteration
 

We wanted our design to reflect the casual, funny, playful, yet at the same time serious, game play style we were after in a non-threatening or overtly sexual manner.

 
Closer to Cherry Pop as it stands now
 

Darker colours were also experimented with to see how that would look.

 
Darker design iteration
 

We settled on a mixture of our second and third design with some further tweaks.

 
Chosen Cherry Pop design
 

Development and Technical Details

Cherry Pop was developed natively on both iPhone and Android using their respective SDKs.

Powered by an ASP.Net MVC3 web service, both clients retrieve and submit answers to a central API in order for the website to calculate the answers to be shown at the end of the game.

Instead of duplicating game play logic (such as calculating the activities or generating questions) in both the iPhone and Android games, the decision was made to centralise this logic at a web server level to reduce development time while also providing the possibility of generating aggregated analytics later.

iPhone Development

While proficient in C#, PHP and other languages, I knew very little about Objective-C development, and invested in a MacBook Pro to get started.

After pushing through a handful of the excellent 193P Stanford iTunes tutorials, I got a bit frustrated from not working directly on my product, so I just fired up xcode and began to develop it and worked out how to do things along the way.

Partial screen capture of one of the Cherry Pop iPhone source files

Several technologies were used to develop the native iPhone application:

UINavigationController Problems

During development I had experienced the same problem as others with UINavigationController's popViewController method.

When you attempt to popViewController (or setViewControllers) during an existing navigation animation, you can corrupt the animation state and leave the UI in a broken state.

I have created and open sourced BufferedNavigationController as a drop-in replacement for UINavigationController, that will queue animations for you and simply make the requested change when the existing one (if any) has finished.

This allows you to not worry about an in-progress navigation when deciding if you can call popViewController.

Target Platforms (iOS 4.2+)

You can specify the minimum target platform when compiling your iPhone/iPad applications in Xcode.

There is a trade-off you need to make between supporting older devices and not being restricted from using newer iOS APIs only available in later versions.

The minimum version worth targeting, as of today, appears to be 4.2+.

Recent statistics from August 2011 suggest 4.2+ minimum will cover ~95% of users:

4.1x looked to have a good chunk of users at 38% around January 2011, but that seems to have changed since August.

Additionally, 4.2.1 is the highest version iPod Touch 1st generation can run, as it is a discontinued product.

So 4.2+ looks to be the sweet spot to target that has the best coverage of devices and API usage.

Unfortunately, compiling for 4.2 (or lower) requires armv6 binary that can increase the size of your App Store upload.

This may or may not be a problem, depending if you mind it pushing you over the 20MB 3G download limit from the App Store (otherwise, customer's must be on WiFi to download your application after the purchase).

iPhone Testing

Testing the application with other people is a little difficult on the iPhone, normally you need to upload the application to individual testers using iTunes and e-mail attachments, which is a bit cumbersome.

I found the free Testflight service to be helpful in distributing new versions to testers.

TestFlight's build management screen

Android Development

Cherry Pop uses several custom button styles that vary in colour, including their push/active states.

Drawing individual Nine-Patch images for each button would have been tedious, and would make adding new images or revising existing ones a slow process.

Instead, we used the 9-patchedit utility to generate the required images automatically after figuring out the initial dimensions.

Since our button's were similar in dimension and only differed in colour styles, it was easy to configure the nine-patch dimensions once and use this utility to generate nine-patch files for every other button.

By choosing to script this operation instead of manually drawing the nine-patch image for each button, we have zero resistance in building a new image or tweaking the existing dimensions, because its trivial to generate the files again.

Website Development

The Cherry Pop website at cherrypopapp.com acts as both a landing page for customers as well as an API backend to support the game itself.

Powered by ASP.Net MVC3 and PostgreSQL, several other technologies helped develop and support the backend platform:

Web Pages: Razor ViewEngine
The Razor view-engine is used to render markup, and is a great improvement over the older WebForms style engine.
Razor ViewEngine with automatic HTML/code detection
NHibernate OR/M
NHibernate, the .Net port of the Java Hibernate OR/M, is used for data access.
Sample OR/M query using NHibernate

NHibernate takes care of the way we load and store entities in the database, and is smart enough to optimise access patterns to improve performance.

Unit testing is also made easier by using an SQLite in memory database for several unit tests.

Database Layer
PostgreSQL was chosen over MySQL as the database platform in order to experiment with new technologies and overcome a technical limitation.

We are using GUIDs for Primary Key values in our database which have several benefits over the traditional auto-incrementing/sequence integer column:

Initially, MySQL was going to be used as the database of choice (having been very familiar with it before), however during performance testing, insert performance suffered as several of the main tables started to grow (500,000+ rows).

InnoDB was used as the the storage engine for each table for transaction support (and only real choice, if you care about data integrity, as MyISAM is not crash safe).

An interesting feature of InnoDB is that it automatically clusters rows in Primary Key order, which does not happen if you use the MyISAM storage engine, or another database like PostgreSQL.
Other database such as SQL Server can optionally cluster by an index.

What this means is that for every inserted row, InnoDB will put the record on disk physically in the right sequential location, moving other rows out of the way to make room if necessary.

With an auto-incrementing Primary Key, where the sequence value is always incrementing (1, 2, 3...), rows are easily clustered in-order because the next row is just appended to the end of the table. Inserting new rows using a sequential, always incrementing integer is relatively cheap.

This is an intended feature, as usually you want to access rows in sequential order (for example, the last 10 orders from a shopping website).

As GUID's are essentially random (even using the GUID Comb generator, which is supposed to create semi-sequential GUIDs), new rows are often not added to the end of the table, but instead must be inserted between existing rows - requiring InnoDB to make space (and re-locate other rows out of the way).

Insert performance dropped rapidly for even small (by today's standards) tables of 500,000+ rows.

An excellent explanation and analysis of this problem is available and I was able to replicate the results in my test environment.

Image courtesy of kccoder.com

It's not possible to turn off the InnoDB clustering, and if no Primary Key is defined, the first NOT NULL UNIQUE index is used as the Primary Key internally by InnoDB.

Solutions included just using an INDEX on this column, and letting InnoDB cluster internally, as well as using the Hi-Lo algorithm, however these solutions were less than perfect.

The Hi-Lo algorithm can quickly grow in the sequence number it will use and there could be 'gaps' depending on whether previously reserved sequence numbers were unused due to the application unloading and forgetting its last known increment state.

While not a big deal, I felt that combined with possibly 'running out' of numbers at some point (even for BIGINT 64bit counters) due to possibly incrementing very large numbers because of intermittent database activity, combined with not being able to easily add new rows should it be necessary manually, without consulting a counter table, I looked into other options.

PostgreSQL was able to maintain consistent insert performance even at 5,000,000+ rows as rows are not clustered by the Primary Key when using GUIDs. Perfect.

Unit Tests
Liberal amounts of unit tests are used in order to ensure basic data access and services are working correctly, as well as game play logic.
Unit tests give us confidence things are working

Test Driven Development practices were followed to a degree - sometimes tests are written before features, but sometime's its quicker to do it around the other way.

ReSharper from JetBrains is a great productivity tool that allows you to refactor with confidence and navigate the code base very easily, and includes extended unit testing support compared to Visual Studio.

NHibernate and Fluent NHibernate
NHibernate is used as the OR/M mapper for data access, and the add-on component Fluent NHibernate is used to describe the domain mapping of entities, instead of the traditional XML files NHibernate supports out of the box.
FluentMigrator

FluentMigrator is used to automate schema updates between development, staging and production by writing 'schema migration' classes to describe schema updates.

This allows for consistent and automated schema changes without any manual (and error prone) work each time another environment such as production needs to be updated.

WebAPI

I found the WebAPI framework (basically WCF for ASP.Net MVC) by Microsoft to very suitable for this project.

It provides a JSON (and XML) access point for the mobile clients to communicate to the backend system, without having to write my own JSON to data class objects mapper.

JMeter for Load Testing
Using Apache JMeter I was able to load test API calls easily while graphing the various results over time. This helped me easily identify the MySQL insert performance issue mentioned earlier, and see how the service would perform under higher than expected load.
JMeter showing the response time of API calls when processing high load
Change IIS startMode to AlwaysRunning

IIS 7.5+ provides a new feature called startMode that allows the initial JIT-compilation of your web application to be performed as soon as IIS is running or a change is detected in your application code.

Normally, startMode is set to OnDemand and means the first request to your website will incur a time penalty while IIS JIT-compile's your application and its dependencies into memory, so that it can serve the request.

This provides a poor experience to the visitor who happens to be the first after an application recycle, either due to worker timeouts, or scheduled application recycling, which occurs by default.

As a work-around to ensure the first request is responsive, you can prevent the application from unloading due to idleness by turning off Application Pool recycling, or have a monitoring service periodically access your website to keep it loaded in memory.

However, by changing startMode to AlwaysRunning IIS will automatically JIT-compile and prepare the web workers as soon as possible, before the first request hits your website.

When using multiple web workers (also known as a web garden) on the same server, each web worker will ensure it has JIT-compiled in time too.

While there is still a JIT-compile penalty across methods being used for the first time, the delay seen by visitors is much smaller because the majority of the JIT work has already been performed ahead of their request.

This provides a much better end-user experience for the first user of your website, because the web application has already been primed and eagerly awaits the first request.

If you update your website regularly, or you have periods of inactivity (and your application gets unloaded or recycled), this ends up being a great way to keep your site ready to respond to requests.

Game API: Idempotent

Our API was designed to be idempotent when possible so that multiple requests for the same action (such as to request more questions, or submit answers) will only be processed once.

Idempotency allows our mobile clients to re-try requests if an error occurs (such as due to a loss of network connectivity) without the chance of breaking the game state by duplicating requests.

Additionally, when players are playing on two separate phones (iPhone only at the moment) with the help of the Bump API, it's possible that both players request the game continue on with the next set of questions at the same time.

By designing with idempotency in mind, the web server can process the first player's request for more questions and automatically discard the second (secondary) request from the second player who happened to also request more questions at the same moment.

Source Control

Subversion is used as the source management system due to the familiarity we have with it.

VisualSVN Server is used as the server repository and is easily setup for use.

Server Provisioning

ASP.Net, and especially MVC3, runs best on Windows servers in my experience, and so several Windows VPS's were purchased to host the infrastructure.

Depending on the popularity of the project, we can adjust our expenses accordingly.

I am using MammothVPS located in Australia for hosting, as I am employed by its parent company, and the feature set is excellent. These servers are not sponsored.

I definitely wanted "developer ops" tools like Linode provides so that I could self-manage when needed.

Server Density is used to monitor the servers and alert us to any problems.

Setting up and configuring a Nagios instance was considered as a free alternative, but I did not think the effort was worth the cost saving.

Pingdom was also considered, but lacked some features such as measuring server performance and memory usage, as it only monitors uptime availability.

I had previously written a plugin to monitor and alarm website availability for Server Density, which I re-used for this project.

Dashboard status of a few servers

Apple Review Process

We were initially rejected from the AppStore a few weeks ago because our product was not in compliance with Apple's review guidelines.

I needed to revise the Apple AppStore version of our project to contain much tamer questions compared to the intended content we wanted to include.

Thankfully, the Android Marketplace does not have such conservative guidelines.

Project Management

While I focused on the website backend (C# ASP.Net MVC3) and iPhone development (Objective-C), my business partner concentrated on Android development (Java).

We would often work remotely from each other and use Google Chat and e-mail to communicate, worked at each other's houses, or would meet up in a park.

As we both still have day jobs, we would work after hours or during the weekends.

Each week or so we would check-in and make sure we were on track for meeting our own deadlines we made for ourselves, and discussing any problems or features we needed to work on.

We would decide to cut features if necessary in order to launch in time, trying to stick to the Minimum Viable Product (MVP) philosophy.

Product Pricing

Cherry Pop was originally going to be a free, ad supported game. We may revisit this in the future, however we reversed our decision and instead went to a Paid App from the start for several reasons.

Marketing

We wanted to avoid spending money on expensive marketing for this project (such as Google AdWords), as I have personally not had much of a return in the past.

We are wanting to see how the project is received after being launched by posting to a few websites we visit and hopefully letting the project stand on its own merits.

We see marketing as being organic, it's something you need to keep working on. We're going to tweak our marketing as time goes on based on the activity we receive.

Initially we were targeting an iPhone release date of 21st December to try and be in the New Releases list across the iTunes Connect shutdown period.

However, my research found a long standing bug with how the New Releases list works in the AppStore.

Your application appears relative to when your application was approved by the review team, not when you activate it for sale. We immediately activated the application as soon as we found out (on the same day of approval), and found our application already buried further down the New Releases list.

It appears that the list is sorted by Approval Date Descending, Application Name Ascending - So you would want to have your application released on the approval date, while making sure the application name is favourably named alphabetically.

Basically, you don't want your marketing campaign to be centered around being in the New Releases list.

We are also promoting the application with friends, a few review websites who are kind enough to review us, and will push for additional marketing based on the reaction's we receive.

Testing Feedback

We distributed the iPhone application to several of our friends and family members in order to get feedback as development progressed.

Notes were taken after collecting feedback, and we discussed any changes we needed to make.

Their feedback was very valuable in shaping the current version, as well as our plans for the future.

Analytics and reporting

During our launch we wanted to track the activity and referrals that were made to our website in real time.

We also wanted to know whether there were any problems customers were having, and how many games were being played so we could get an insight into the product's popularity and sales as soon as possible.

Real-time website analytics

fnordmetric was recently released which looks great, however setting it up properly would take more time than I wanted to spend at that moment.

Google Analytics released a new real-time feature to their analytics dashboard, which is exactly what we wanted so that we could see how much interest we would receive during our launch.

Google Analytics Real-Time dashboard

You can find this feature in your own Google Analytics account by visiting your website's dashboard (you must be using the new account theme) and then clicking on the Home -> Real-Time menu button.

Internal reporting statistics

We created a simple web page on our end to indicate the number of games that have been played so far. Pretty basic, but it gets the job done!

Error handling

Any API or website exceptions are logged to a central database for inspection. This will allow us to resolve errors before they become a problem for more customers.

Logged errors and messages for review

We can mark errors in our code a severity rating high enough to raise the alarm with our monitoring system to expedite an error getting investigated.

Conclusion

I hope you enjoyed finding out more about the development process of Cherry Pop.

If you'd like to check the application out for Android or iPhone, visit cherrypopapp.com.

- Andrew Armstrong (andrew@scramblemedia.com.au)

Tame Questions Only
 
Explicit Questions

HackerNews or redditor?

If you happen to use any of these sites and think its worth up-voting or commenting, please do so, it's appreciated.

Discuss or upvote on HackerNews Discuss or upvote on reddit