API Centric Design with Django and GAE

I wasnt really sure what to call this post.  My current situation is I write django apps (well so many words within that sentence can be loosely defined).  I guess after playing with django for last two and a half years (and Google AppEngine, GAE, ever since it came out) I am at a very interesting (or frustrating) crossroad.  What platform to choose and stick with – appengine or django.  Actually I had this conundrum about a year ago and I have essentially postponed this choice by what I call API centric design (ACD).  I am pretty sure ACD is just another fancy acronymn for existing techniques out there but still bear with me.

Django can be quiet easily used with GAE.  And infact that is what I have done.  I use django entirely within GAE.  The real impedence mismatch pops its ugly head thanks to the completely different backend schemes used by GAE and Django (key-value store vs relational DB).  As much as I am not a fan of the GAE, it is a pretty good platform for prototyping.  Well its free and hence, given the state of the competition, the best choice (for now) in the price range.   So how does one use GAE for prototyping an application, leaving it out there to collect feedback (and failing early and failing often) and move the application a platform of choice when the limits are reached and the constraints in GAE are just too much?  A redesign of an app to leverage non-GAE environments is certainly not the way to go.

What I have done is only expose an API from the app point of view.  Hide the back-end specific details behinds its own models.py (django uses models.py to hold the data models for an applications).  Its actually a lot easier than it sounds (if it already sounds easy, then it is even easier).  What is trickier is that one would have to exercise considerable discipline in following this.  Every call in the application should call only the exposed api instead of using the data-backend related calls provided by the platforms (GAE or Django or anything else).  Even leave the implementation and the optimisation to the specific backends.

As an example, let us take the famous “counter” example.  A typical way to have counters in traditional RDBMs based platforms is to have a table that stores the rows of name/counter pairs.  For DBMSes that are strong with write’s this is great.  GAE however recommends a different approach using shards.  Check Brett Slatkin’s video on this.  So how do you choose between these two?

Firstly settings.py needs to have a variable called USING_APPENGINE (or something similar) that will be set to True if the app is running in GAE and False otherwise (ie a native django app running on the test server or on other webservers).

Suppose our counters api is as follows:

def get_counter(counter_name)

def incr_counter(counter_name, byhowmuch = 1)

Now il need 3 files for the api – counters.py, djcounters.py and gaecounters.py.  Only counters.py will be imported in any other part of the application requireing counters.  djcounters.py will have the get and incr counter implementations for traditional RDBMBs and gaecounters.py will have the same for a GAE specific implementation.

counters.py will be:

from django.conf import settings

if settings.USING_APPENGINE:

from gaecounters.py import *


from djcounters.py import *

Thats it.  In your calling app simply do:

import counters


counters.incr_counter(“xxx”, 10)

This is only the beginning.  Clearly this extends to other types of backends as well (ie support TokyoCabinet in the future etc).  When writing an application you shouldnt have to worry about the optimisations.  Nothing stipulates that gaecounters.py will have to implement Brett Slatkin’s stragey when using counters in GAE, but having separate files allows for this optimisation in the future.  Let the api abstract it all away.

Ofcourse it may appear that this will explode the number of functions required in the API.  Not necessarily (ie functions having to bepeated to do filters or object creations or deletions).  Some of the common patterns are object creation, fetching an object by ID or by certain attrib/column values, deleting records.  There are others, but these are the most common ones.

So I have a base djhelpers and gaehelpers file which define object creations, deletions, searches for ANY kind of object class and each one is optimised (sort of) to that specific implementation.

For instance, in django objects can be created with the method:

def create_object(obj_class, **kwds):
    return obj_class.objects.create(**kwds)

while the GAE implementation would be:

def create_object(obj_class, **kwds):
    obj = obj_class(**kwds)
    return obj

Note these are only basic ones.  Other implementations could go through the kwds list and get the parent and id parameters to set these on the object as required.  Alternatively, you could only save optionally and so on.

So an object creation in another part of the app is simply a matter of doing:

create_object(MyObjectClass, a = 1, b = 2, c = 3)

and the object is created without having to worry about the backend type.

This is again a simple way of hiding the implementation details.  The same could be implemented using metaclasses and properties to define a DBObject class whose metaclass would be the implementation specific class for the above and other object specific methods.


3 thoughts on “API Centric Design with Django and GAE

  1. APIs and layers of abstraction … sounds like your yearning for a bit of Java!

    The Java AppEngine impl makes code reuse much easier (e.g. using JPA) which is a very much a Java way of doing things. However, its a slipper slope and next think you know you’ve truckloads of XML & boilerplate. Of course the alternative solution is to hide it all away and do things automagicly, but people dont like magic either.

    Is there a middle ground?

    • Actually I am not against java. But I think after doing Python, I find Java too verbose and unnecessarily clunky. While java has good reflection capabilities, doing what I am doing with GAE would be too messy dont you reckon? Never used JPA, what is it like? And also somehow I have this wierd feeling inside me when you are “given” all this boilerplate and xml stuff to play with (my exception to that is how it is done in android :D)… i like magic, only when i can sort of know what is happening…

      By the way what I wrote in this one is actually common-sense, nothing smart or special. Question is at which point does it go from “hiding the innards” to “rewriting the entire thing”?

      * I should add that it is not so much about abstraction rather than “guaranteed behaviour” that I am after. I know its a blurred line, but an important one nonetheless. Also I dont want layers. I just like little components that I can retrofit as I need them…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s