Django/Appengine Coexistence, Part 1 – Environment Specific

We’ve heard it all.  Appengine has uber scalability and that too on demand.  And it is based on Django.  Clarification – the python based SDK is based on django.  And yes it is a great environment to develop in.  No issues.  But the thing that still causes me a bit of concern is the datastore.  Oh it is fantastic, scalable and all that.  But the limits are still limits.  (I havent even mentioned the 1000 limit on blobs and files which is another big issue).  And also my main concern is if I want to commit myself to writing all my code for appengine and then be in a position where porting to django would involve a total rewrite. Ok there are projects like appenginepatch and so on, but to me its trying to mold one paradigm into another.  Why patch a key/value pair based datastore into an sql backend when each has their own strengths and you are better off writing interfaces that do it in specific ways so the same app at run time will choose the right implementation based on the environment rather than all the adapting of one paradigm to another?

So anyway, my M.O is to have a single project with the view and controller logic that is common for django and appengine but to have seperate model classes and APIs targetted to each backend.  The models are accessed via the API classes that do all django/appengine specific magic behind the scenes transparent to the views and the controllers.  I have talked more about this in a previous post.  However I did not provide a specific implementation.  This is now open sourced as part of the Djapps project.  At the moment this is still in its infancy but it is a collection of app to do a lot of common tasks like external authentication, provide model accessors transparently based on appengine or django and so on.  Still evolving and I will talk about this in future posts.

But to get started with all this, the first thing that needs to be worked on is  For better or worse, this file is where all your app specific settings go.  So what I do is literally have a that i import from and have environment specific  So I include one depending on what kind of environment I am on.  Environments for me would be “appengine dev server”, “appengine production server”, “local django server”, “production djanso server” etc.

So as an example, here is my  Very stripped down:

# -*- coding: utf-8 -*-
import os
import djapps

# import all common settings - we will override them later on
from djapps.default_settings import *

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'xxxxx'

# The root folder of the project
PROJ_ROOT           = os.path.abspath(os.path.split(__file__)[0])

 # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
 # Always use forward slashes, even on Windows.
 # Don't forget to use absolute paths, not relative paths.
 PROJ_ROOT + "/templates/"

import dev_profiles
# from dev_profiles.profile_dev_gae import *
from dev_profiles.profile_prod_gae import *
# from dev_profiles.profile_dev_nongae import *
# from dev_profiles.profile_prod_nongae import *

if not USING_APPENGINE: ROOT_URLCONF = 'myproject.urls'

Now in the above dev_profiles is the folder where all my different environment based settings files are (I call them profiles).   So an example env specific settings py (profile_prod_gae from above) is:

SITE_URL            = ""
SITE_NAME           = "myproject"
SERVE_STATIC        = False
USE_TEST_LOGIN      = False
DEBUG               = True
PROJ_STATIC_HOST    = "/static"
DATABASE_NAME       = 'myproject'
DATABASE_USER       = 'xxx
DATABASE_HOST       = ''
DATABASE_PORT       = ''
FORMAT_PARAM        = "format"
EMAIL_HOST          = "localhost"
EMAIL_PORT          = 25
 # 'djapps.auth.external.middleware.MultiSiteAuthMiddleware',

Thats it.  Nothing escapes customisation this way.  In case y ou are wondering.  The is just the that comes when you first create a django project (with startproject).  Atleast it is very close.

Mind you this has served me very very well on past projects and has not required any major rewrites.  One thing that does annoy me is the last line where I have to manually set the settings I have to choose (ie by uncommenting the line).  You can get around this by checking for an environment variable and importing one based on that but again just a personal preference.

Let me know how this goes for you.


8 thoughts on “Django/Appengine Coexistence, Part 1 – Environment Specific

    • Cool. Haven’t had a chance to look at it close but certainly looks promising. I usually feel that models are best customized to the platform. For instance let’s take a Twitter style application. With SQL joins are there and the easiest solution. But with appengine the way to do it is to actually use string list properties and then do a bulk fetch from the datastore. Ok I’m trivializing a lot of the solution but thats the gist of it. Brett slatkins goes into the details. But i digress. Point I was making was joins in SQL are standardized constructs (again not arguing about the optimality). But stringlist properties are very appengine specific. So firstly how does the orm handle this across multiple model paradigms? Especially for something that is so app specific. Eg a dumb orm would treat handle the Twitter application by dong the join manually and then fetching records individually from the datastore.

      I am hoping newer Orms could recognize patterns in your models than just models as classes and trying to adapt one backend into another. That is why I prefer doing the model individually for each backend and exposing only the functionality that is expected of the models. Maybe a model pattern language is what is required instead of just a query langage!!!

      • For now, we’ll provide a ListField which only works on nonrel backends. This could be emulated on SQL via different tables, but it wouldn’t be ideal. The current goal is to at least be portable across the non-relatinoal/NoSQL databases, so we have a common standard.

        We also thought about something like the model pattern recognition that you talked about. This would indeed allow to use the same code for all backends (including SQL). If you want to implement this we can discuss this further on our group:

      • Mate I would love to help. Time would be an issue. But let is talk more on it. What I started doing with djapps was have a model layer that had abstracted methods for simple patterns like object creations deletionss saves etc but left specific patterns like the Twitter style joins to apps using them. But it does require a bit of thinking. Another problem is the variety of limits each platform poses.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s