A Mercurial Branching Workflow

The StreetHawk code is broken up into multiple repositories, the backend webapp, the iphone UI, the iphone API layer that accesses the database and handles all logical/operational aspects of the iPhone app, the android app (with the same breakdown), tools etc.  Up until now all code has been checked straight into the “default” branch (HEAD).  Going forward we have decided to adopt the following scheme inspired by and based on workflows prescribed by Steve Losh and Vincent Driessen.  I would particularly recommend Steve’s tutorial on branching which was very very critical in helping me understand the general branching techniques in mercurial.  If anything this post is a summary of their article to act as a “manual” for ourselves and is prone to change given all the nasties we may find in the future.

Command Summary

Firstly a recap of some of the branching and tagging commands that we will use later on.

Creating a new branch

# Get into the branch from which the new branch is to be created.  
# This would usually be "default" if it is a new future development or "stable" if a bugfix
hg checkout <parent_branch_name>
# Create the new branch
hg branch <branch_name>
# commit it - at this point it is a local branch
hg commit -m "Added <branch_name> branch"
# push it to the server to create a remote branch
hg push -b <branch_name> --new-branch

Merging from another branch

Time to time you may find the need to merge from another branch.  This is simple.  This will ensure the source_branch is merged into the target_branch.  NOT the other way around (even after you do a push).

# Get into the branch "into" which you want changes merged 
hg checkout <target_branch>
# merge from the source branch
hg merge <source_branch>

Closing a branch

Some times (quite often actually) you may find branches created unnecessary.  To remove the branch:

# Get into the branch you want to close (by now you can see the general pattern here!)
hg checkout <target_branch>
# close it
hg commit --close-branch

Tagging a branch

Tag a branch on every release.  Simply do:

# Get into the branch you want to close
hg checkout <branch_name>
hg tag <tag_name>

The convention used here is: <PUBLIC_VERSION>_<PRIVATE_VERSION>

PUBLIC_VERSION = <MAJOR_VERSION>_<MINOR_VERSION>_<BUGFIX_NUMBER>
PRIVATE_VERSION = <PURPOSE>_<BUILD_NUMBER>
MAJOR_VERSION - indicates a back incompatible version
MINOR_VERSION - new features added
BUGFIX_NUMBER - bugfixes updates
PURPOSE - purpose of the build - eg "APPSTORE", "TESTFLIGHT", "ADHOC" etc...
BUILD_NUMBER - The build attempt for a given PUBLIC_VERSION and purpose.

Branching and Release Workflow

In this scheme we have the following branches:

“default” – This the dev branch that contains all the checked in features and bugfixes that are *ready* to go into the next stable release.

“stable” – The branch that tracks the stable production releases.  Code from “default” is always merged into stable before a release.

“feature_xyz” – New features are added and developed on their own branches (eg feature_camera, feature_location etc) and merged into “default” upon completion but NOT before they are ready to be accepted into are release.  So only merge these into default when they are ready for the next release.

“bugfix_xyz” – Similar to the feature branches but these are usually branched from stable and then merged back into both “default” and “stable”.  These are usually urgent changes/fixes to be applied on released code.

Roles

Given the above branches, an appropriate workflow would contain the following roles (could be the same person):

Developer - Is responsible for adding new features and bug fixes and most likely would be adding to “default” or “feature” branches (or onto stable via bugfix branches).

For example to add the new “awesome” feature, one would do:

# get into default
hg checkout default
# do an update
hg pull -u
# create the feature branch
hg branch feature_awesome
# commit it - at this point it is a local branch
hg commit -m "Added <branch_name> branch"
# add the awesome feature
# do a few commits A, B, C
# push it to the server to create a remote branch ONLY 
# when ready for release
hg push -b <branch_name> --new-branch

Release Creator/Owner – Creates new stable production releases.  Note DO NOT do a rebase here as that would detach local changes.  We specifically want to keep track of all past activity.

# get the latest on all branches
hg pull -u
# get into stable
hg checkout stable
# Merge default into stable to get all production ready 
# features into stable
hg merge default
# Do the testing and when time to release, tag it (you are still in stable)
hg tag <tag_name>
# push the new tag
hg push

 

Detecting race conditions in iOS

Been working on my new startup – Streethawk (formerly known as Dealsta) for the last four months.  Actually Ive been working on it with Dave since last year but jumped on it full time pretty much the day my beautiful baby girl – Shivani Gayathri – was born.  I think in a way she showed me how much of a wuss I was for not doing something I was passionate about.  But that’s another story.

So I jumped on the iOS client for our app and have been doing pretty much that full time for the last  six months.  An amazing ride and have learnt so much I cannot beleive it.  50 KLOC in four months does that to you!  But there’s still so much to learn.  Hardly enough time.

One issue I wanted to address was race conditions.  Nothing magical to it and this is just one approach.  iOS can allow one to lock access to a resource via the @synchronized keyword or via the use of  semaphore objects (dispatch_semaphore_t).  Excessive locking just imposes additional overhead causing possible “hangups” on the UI/main thread giving the jerky feeling (With dispatch semaphores a kernel level trap only happens if indeed there are multiple threads waiting on the resources so in a sense dispatch semaphores are cheaper and faster than locks based on @synchronized but thats a very very simplistic explanation).

<Digression>

I am a big big user of blocks (and GCD) for my iOS apps.  Infact most of my style with asynchronous development has been favoured heavily towards blocks dispatched to async queues.  Heck I even wrote a Request class wrapping NSURLConnection that would do everything via callback blocks rather than using delegates that was just too messy and spread out everywhere.

The general way of using dispatch queues in GCD is to do something like (nothing fancy here):

// create a serial dispatch queue (from OSX Lion onwards you can also create Parallel queues)...
dispatch_queue_t    my_queue = dispatch_queue_create(queue_name, NULL);

dispatch_async(my_queue, ^{

// do the work here in the queue.

});

Now what is interesting with all this is that iOS takes care of how many threads to create across all your queues and manages the scheduling of the tasks across the queues.  There are some interesting memory management facts you need to know about but you are fine otherwise.  There are a few advantages to this (amongs others):

1. You migrate away from manually having to managing threads and scheduling.  The OS handles how many threads to create across the several queues and manages the scheduling across the queues.
2. You replace locks to enable exclusive access to a resource or a critical section because (with serial queues) only one task is run from each queue at a time (also constrained by how many threads the OS creates).
3. The code actually looks more “continuous” spatially even though they are not temporally continuous.

<End Digression>

Back to it. Regardless of which approach you pick, concurrency risks race conditions.  So what you really need is a way to detect when multiple threads are accessing a particular section and instead of locking access to other threads, do a log or even better an assert triggering the dumping of a call stack.

For this I have created an RCTool class that simply has the following 3 primitives:

test_begin_lock(const char *filename, const char *function, int line, int mode, NSString *resname, BOOL alreadyLocked)

What this does is creates an entry to mark that the current thread (from which this method is called) is recorded as a “requestor” of a lock at this point in time.

The key parameters here are “mode” and “resname”

“resname” is the name of the resource being locked or checked – and must be unique to mark the resource being verified – very very app and developer specific.

“mode” controls what kind of access is required for the thread entering this section and/or accessing the particular resource.

There are three scenarios here:

1. If there are no other threads holding a lock at this point in time, ie no calls to test_begin_lock with a matching test_end_lock (see below), then the current thread is now the “starter” of the lock at this section.
2. If there are other threads holding locks at this section this is where the “mode” parameter comes in.  The mode parameter is either WRITE (ie only one thread – the current thread – is allowed beyond this point) or READ_WRITE (ie multiple READ threads are allowed but only the first WRITE thread and no subsequent threads of any kind are allowed).  So the following are possible:
   1. If another thread has a WRITE lock (and only one such thread can exist) then this method asserts.
   2. If there are other READ threads and this thread is seeking a READ lock, this thread is added to this group of threads and allowed to proceed (ok to have multiple readers).
   3. If there are other READ threads and this threads is seeking a WRITE lock, then the method asserts.

Assertions in this methods are hints towards which parts of the code have possible conflicts and are good candidates for re-factoring (or as I call it temporal factoring) into queues or other synchronisation methods.

test_end_lock()

This method marks the end of a critical section for  a particular thread (which was begun with the previous method).  Note that there could be multiple threads “alive” due if all threads were READ etc.

Now this is no substitution to one of the many static or runtime analysis tools that costs a few arms and a few legs.  But what this tool does is lets you run your code normally and *before* the first race condition occurs rather than somewhere down the track long after when the data corruption has occured.

Secondly this is a debug only tool.  Naturally one would use this tool/method/class to detect possible race conditions and critical sections so they can *fix* it rather than just leaving it there.  So this doesnt make sense on production.

HOWEVER, it doesnt seem useful or easy to have to add and remove this repeatedly across several parts of your code depending on production or not.  So I use a couple of macros that allow all this stuff to be easily turned on/off and also to automatically pass the function, filename and line number automatically.  It is something like this:

#define LOCK_NO_WRITE       1
#define LOCK_NO_READWRITE   3 
#if IN_PRODUCTION
#define BEGIN_LOCK(mode, res, ...)
#define BEGIN_SYNCHED_LOCK(mode, res, ...)
#define END_LOCK(mode, res, ...)
#else
#define BEGIN_LOCK(mode, res, ...)            \
    test_begin_lock(__FILE__, __FUNCTION_NAME__, __LINE__, mode, [NSString stringWithFormat:res, __VA_ARGS__], NO)
#define BEGIN_SYNCHED_LOCK(mode, res, ...)    \
    test_begin_lock(__FILE__, __FUNCTION_NAME__, __LINE__, mode, [NSString stringWithFormat:res, __VA_ARGS__], YES)
#define END_LOCK(mode, res, ...)              \
    test_end_lock(__FILE__, __FUNCTION_NAME__, __LINE__, mode, [NSString stringWithFormat:res, __VA_ARGS__])
#endif

So to obtain a READ lock simply call:

BEGIN_LOCK(LOCK_NOWRITE, @"resource_name")
// do things here that has read only access to this the resource
END_LOCK(LOCK_NOWRITE, @"resource_name")

And to get a write lock (to prevent other readers as well):

BEGIN_LOCK(LOCK_NOREADWRITE, @"resource_name")
// Here the resources is also being written to
END_LOCK(LOCK_NOREADWRITE, @"resource_name")

Thats it.  In dev simply set IN_PRODUCTION to zero and off you go.  You will see this slowing down your phone or simulator but at least its towards a good cause!  The actual implementation is attached here.

 

Cyginstall – The Stateful Cygwin Installer

Hands up those who have tried installing cygwin on windows only to have the damn installer not completely downloading sources or you found that by selecting multiple mirrors, the installer would just get stuck on a slow or buggy mirror?

Well back up a bit.

Hands up those who were *forced* to use windows and hence had to install cygwin?

Me and me again!

At work I am on a windoze box while having to ssh into a linux box to do my development (let us not go there).  Luckily cygwin has helped me in the past but only because I managed to complete the installations out of sheer luck.   For some reason (may be limited intelligence) I could not select the fastest or the most reliable mirror and the installations kept bombing out after hours of what seemed like progress.  Seems wierd but I would have thought the cygwin mirror seeing what files it had already downloaded would have helped.  But no such luck (was it some option I missed)?

So I decided to write a downloader myself (you would still have to call the cygwin setup tool to install the sources).  It HAD to be stateful.  You interrupt it at any point and it should continue where it left of.  So out of that came cyginstall.

It also has a few other goodies:

1. Multi threaded (defaults to 5)
2. Web based UI
3. Easily select or deselect mirrors
4. Works through proxy with NTLM authentication.
5. Better reporting of download status and mirror health.

So please give it a go and let me know what you think.  I managed to do a whole download but that does not mean there are no bugs so would appreciate it if you could let me know of any and all issues that need fixing.

Also the UI is very drab.  Very basic but drab, so any help or advice there would be gold.

 

2010 in review

Have to say. Pretty impressed with wordpress’s “summary” feature/email/view etc..  I wonder if me lazily posting this back as a post actually counts as a post?  In which case this is my first for the year – happy new year all!

———————————————–

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads This blog is doing awesome!.

Crunchy numbers

A helper monkey made this abstract painting, inspired by your stats.

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 4,000 times in 2010. That’s about 10 full 747s.

 

In 2010, there were 5 new posts, growing the total archive of this blog to 42 posts.

The busiest day of the year was March 31st with 41 views. The most popular post that day was Android Hack – Parcellable in eclipse.

Where did they come from?

The top referring sites in 2010 were anddev.org, rainbowdevil.jp, markmail.org, mail-archive.com, and mindtherobot.com.

Some visitors came searching, mostly for gray code generator, django app engine, google maps captcha, django mobile, and aidl file.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

Android Hack – Parcellable in eclipse March 2008
18 comments

2

Django/Appengine Coexistence, Part 1 – Environment Specific Settings.py February 2010
7 comments

3

Mango – Mobile Django – Templates March 2010
2 comments

4

Gray Code Generator October 2009
2 comments

5

Carnatic Violin March 2007
17 comments

 

New Start

Well it has happened on wednesday.  I have now started work with Macquarie bank.  Pretty excited (though a bit scared).  WMS was awesome and despite my initial disbelief I was fortunate to meet some of the smartest and, more importantly, the most passionate people I have ever met.

Wish me all luck folks.  Certainly a bit of a pain to be wearing business-y outfit to work every day but hey its small price!

 

DocTest++ – Extracting test cases embedded in Source Files

UnitTest++ is brilliant in how it lets you structure and manage your tests cases but it still requires a “focus-switch”.  As I am writing a function, I should be writing a test for it, but going to another file (or compilation unit for the technical term) makes me loose focus of where I am and what I was doing.  Clearly I am only mortal and this doesnt apply to the uber coder gods out there.  And this is where python’s doctest is awesome.  Then and there test insertion (yes they are validated at a different phase but atleast you can see what tests are around).  Not to mention more info on the documentation.  After all tests cases as documentation are a pretty powerful idea.

So it hit me.  Why not doctests for C++.  Lo behold DocTest++.  Wrote this last weekend because out of laziness I was’nt really adding test cases to my mango code, as much I should.  Mango certainly does have a few unit tests (and growing rapidly) and even more certainly has no plans to replace those unit tests.  But why not doctests as well?  So DocTest++ extracts tests embedded with C/C++ code and creates test cases for them.  More at the project page on git hub for how it works and what it does.

And to go a step further, this is part of the build process in my make file. A target XXX.test.cpp is (automatically) added for file XXX that contains the @test tag.  This gets compiled along with all tests.  More importantly if any of the files change or new @test tags are added, these are picked by the build system.  Wooot!

Any improvements or feedback?

 

Mango – Mobile Django – Templates

I am a strong believer in a future where mobile devices would actually be servers too.  This is not a new idea or anything.  And Il admit I jumped on the bandwagon quite late.  Though jumping is not what I did out of trying to follow the popular opinion.  It all started when I got the iPhone and it hit me like a truck.  So it wasnt a bandwagon jumping moment as it was a getting-hit-by-a-truck moment.

Anyway, clearly servers on mobile devices instead of server boxes sitting in some rack is exciting.  Imaging you are on the move with a few people on a train and voila you have a webserver you can put up for sharing.  Mind you, this is not that different or even superior to P2P (locally via Bluetooth or globally).  Just another alternative.  So one of the tools I figured Il use quite a bit on my mobile, should it take off as a webserver is a templating engine.  Here is where Mango comes in.  This was actually inspired by Django templates.  And being a big worshipper of Django, I think of Django as a philosophy and not just as a framework.  So first thing was the port of the templating engine.

Where exactly am I porting it to?  First of all the code is opened under an Apache license and is on git.  Secondly I am porting it to android and objective C.  The objective C version is woefully behind, where as the android version has support for a lot of stuff (for, if, include, variables etc) and has a full set of unit tests with it.  Frankly I find android a much much more intuitive platform to develop than XCode (and objective C).  May be it has to do with my familiarity with java.  The objective C version will follow shortly.

So why another templating engine – Well I had envisaged 2 use cases for it:

1. Consume json or xml data from a backend and apply a template on the data from the client (mobile).

2. Mobile generates the content and applies a template to it before sending off the response to a client’s request.

Either way a separate templating engine helps.

Please feel free to try it out. Would appreciate any and all feedback and not to mention help with extending it to a large number of tags and filters.

1. Check it out from git at http://github.com/panyam/mango.

2. Start eclipse (must have java and android)

3. Open the mango and the mangotest projects (from mango/android and mango/android/test respectively).

4. Select the mangotest project and Run As Junit – watch the unit tests.