I wasnt really sure what to call this post. My current situation is I write django apps (well so many words within that sentence can be loosely defined). I guess after playing with django for last two and a half years (and Google AppEngine, GAE, ever since it came out) I am at a very interesting (or frustrating) crossroad. What platform to choose and stick with – appengine or django. Actually I had this conundrum about a year ago and I have essentially postponed this choice by what I call API centric design (ACD). I am pretty sure ACD is just another fancy acronymn for existing techniques out there but still bear with me.
Django can be quiet easily used with GAE. And infact that is what I have done. I use django entirely within GAE. The real impedence mismatch pops its ugly head thanks to the completely different backend schemes used by GAE and Django (key-value store vs relational DB). As much as I am not a fan of the GAE, it is a pretty good platform for prototyping. Well its free and hence, given the state of the competition, the best choice (for now) in the price range. So how does one use GAE for prototyping an application, leaving it out there to collect feedback (and failing early and failing often) and move the application a platform of choice when the limits are reached and the constraints in GAE are just too much? A redesign of an app to leverage non-GAE environments is certainly not the way to go.
What I have done is only expose an API from the app point of view. Hide the back-end specific details behinds its own models.py (django uses models.py to hold the data models for an applications). Its actually a lot easier than it sounds (if it already sounds easy, then it is even easier). What is trickier is that one would have to exercise considerable discipline in following this. Every call in the application should call only the exposed api instead of using the data-backend related calls provided by the platforms (GAE or Django or anything else). Even leave the implementation and the optimisation to the specific backends.
As an example, let us take the famous “counter” example. A typical way to have counters in traditional RDBMs based platforms is to have a table that stores the rows of name/counter pairs. For DBMSes that are strong with write’s this is great. GAE however recommends a different approach using shards. Check Brett Slatkin’s video on this. So how do you choose between these two?
Firstly settings.py needs to have a variable called USING_APPENGINE (or something similar) that will be set to True if the app is running in GAE and False otherwise (ie a native django app running on the test server or on other webservers).
Suppose our counters api is as follows:
def incr_counter(counter_name, byhowmuch = 1)
Now il need 3 files for the api – counters.py, djcounters.py and gaecounters.py. Only counters.py will be imported in any other part of the application requireing counters. djcounters.py will have the get and incr counter implementations for traditional RDBMBs and gaecounters.py will have the same for a GAE specific implementation.
counters.py will be:
from django.conf import settings
from gaecounters.py import *
from djcounters.py import *
Thats it. In your calling app simply do:
This is only the beginning. Clearly this extends to other types of backends as well (ie support TokyoCabinet in the future etc). When writing an application you shouldnt have to worry about the optimisations. Nothing stipulates that gaecounters.py will have to implement Brett Slatkin’s stragey when using counters in GAE, but having separate files allows for this optimisation in the future. Let the api abstract it all away.
Ofcourse it may appear that this will explode the number of functions required in the API. Not necessarily (ie functions having to bepeated to do filters or object creations or deletions). Some of the common patterns are object creation, fetching an object by ID or by certain attrib/column values, deleting records. There are others, but these are the most common ones.
So I have a base djhelpers and gaehelpers file which define object creations, deletions, searches for ANY kind of object class and each one is optimised (sort of) to that specific implementation.
For instance, in django objects can be created with the method:
def create_object(obj_class, **kwds):
while the GAE implementation would be:
def create_object(obj_class, **kwds):
obj = obj_class(**kwds)
Note these are only basic ones. Other implementations could go through the kwds list and get the parent and id parameters to set these on the object as required. Alternatively, you could only save optionally and so on.
So an object creation in another part of the app is simply a matter of doing:
create_object(MyObjectClass, a = 1, b = 2, c = 3)
and the object is created without having to worry about the backend type.
This is again a simple way of hiding the implementation details. The same could be implemented using metaclasses and properties to define a DBObject class whose metaclass would be the implementation specific class for the above and other object specific methods.