Google App Engine

Well ive been spending the last week and a bit with my first AppEngine project.  I had recieved my account on the day it was released (yes a shameless boast there )…  Unfortunately due to exams never got around to doing anything with it.

So as a simple project I figured Il write (yet another online) chess game – WizChess.  Note that there is already a Blitz Chess as part of the appengine samples page.  I wanted to write one just to get a feel for the engine and to largely to learn a lot of things myself.  There are a still a few things I have not implemented (eg on a castle, the server updates, but the client does not properly update UI.  en-passante-s are not implemented.  Timed game not hapening yet, chatting between players not yet happening and so on).

It was a very interesting experience.  The documentation was quite nice and full, but still I was not fully satisfied.  For some reason I found django a lot more satisfying as a web-app dev platform.  For the models for the game I had chosen to follow a more django style (relational) rather than harnessing the rich model extensions provided by appengine (mainly the Expando Models).  I had done this as I wasnt sure if I wanted to seal my app purely to the AppEngine platform.  And after finishing the project, I am still not sure that I do.

I had got a whole bunch of puzzle games from here.  Now I had needed a way to update the database with an initial data set.  So my way was to have a url on the webapp (something like /resetdb) that would load the initial board templates (and in the future games that are saved and so on).  However, since this loading would take some time, I had to resort to breaking this up into several parts and calling each of these individually.  eg:  Id nomally have:

class ResetDB(webapp.RequestHandler):
    def get(self):
        for board in my_boards:
            add_board_to_db(board)

Now as the number boards in my data set increased, the time to put them in the data store would naturally increase.  But there seems to a time limit on each request, which would make my request fail.  So I had to resort to writing a client side script that would call ResetDB with parameters (eg, firstboard and lastboard) and then also break em up as more saved games were restored etc.  Absolute Pain!  Its all good for appengine to be smart about things but to impose assumptions severely limits the usefulness of this platform and I hope in the future this is released.

Next thing was the model apis.  The Expando model is awesome, but once again I was not happy about the lockin.  But this was not my main concern.  The querying api is extremely poor.  Django has amazing querying facilities where one can specify pretty much arbitrary query filters (not a and c or a minus x and so on).  AppEngine does not support “not”s in its query.  Aaaaaaaaaaaaaaaaah.   Also I couldnt find an easy way of chaining queries.  Eg, to find a game, where either player0 or player1 (my way of saying white or black) was user, I had to do this:

games_with_user = []
games = models.Game.all()
games_with_user.extend([g for g in games if g['player0'] = current_user])
games_with_user.extend([g for g in games if g['player1'] = current_user])

Yuck!  Same thing in Django happens with:

games_with_user = models.Game.filter(Q(player0 = current_user) or Q(player1 = current_user))

(ok I did that syntax from memory so it may not be right, but it is something similar).

Now the final that REALLY annoyed me was lack of Comet (or any reverse ajax) support.  Essentially I poll the server every 5 seconds to get updates on whether the opponent has played.  Would have been great for the server to notify the client instead of having to keep polling and waste requests (I believe there is a limit of 6.5 million requests a day or is that 650K requests?)

But having said all this, the main (only?) advantage of the platform is the fact that we dont have to worry about scalability.  Unfortunately this is a HUGE advantage.  Normally I could write an app, and host it on my own boxes at home (if i want it to be free), but then il soon be hitting limits with 10 users and Il have to find a hoster and also handle scalability issues myself.  So migrating my own code to appengine at THAT point would be quite a nightmare.  By writing and prototyping it on appengine allows me to fail fast.  Looking at the pricing, app engine would charge about $40 per month per app (with reasonable traffic).  If my app ever got to that stage, I could always port it out of appengine (as long as I dont use Expando and stick to django style object modelling – with which I dont loose any power).

Please let me know if you would like the sources.  I am not sure how to upload files to wordpress here.

Also I would really appreciate it if you could give me feedback on my learning here.  As a python and web-app noob, Id be grateful for pointing any errors in my ways

C++ – Python Bindings

Version 3.0 of omnifs saw the onset of python bindings for the omnifs library (written in C++ using Curl). In this post I am going to talk about what I had learnt in writing python bindings for C++. What this means essentially is that a library written in C++ can now be used from python. This is ideal for situations where the core modules can be written in a more efficient/lower level languages and can be controlled in a higher level scripting language. This, some (including myself) believe would assist in testing and development in general.

I will be referring to libomnifs, part of the omnifs package, in this posting for any examples. I wont be showing the details of libomnifs, and will focus on what additions were required to enable python bindings.

I have used the Boost.python library for creating the python bindings to omnifs. There other ways of doing this, but I wont go into them here. I had chosen Boost.python as it was fairly well documented and easy to follow (plus it looked cool).

Prerequistes

1. First download and install the boost library. I had used version 1.34.0. Please follow the installation guide for Boost on how to do this.
2. Then read the bjam tutorial. bjam is the build system created and used by Boost. It is very very powerful and can make life a lot simpler (once you get the hang of it).
3. Read the boost.python tutorial – This is a must. In fact this highlights everything one would need to create the bindings. I will only document what “extras” I have learnt in the process and will try to keep the repititions to a minimum.
4. Install python if you do not already have it.

ONCE AGAIN – The boost.python tutorial is fantastic so please have a look at it.

Step 1: First Make a list of Exportable Classes

In my case, one of the classes I had chosen to export was OMNI_Action. OMNI_Action class is superclass of all actions that can be taken agains/on the omnidrive server. For example “ListFolder”, “UploadFile” etc.

The public interface of the OMNI_Action is (protected, static and private members are not shown):

class OMNI_Action
{
public: // Virtual methods and constructors
OMNI_Action(OMNI_Session &p_Session);
virtual ~OMNI_Action();
virtual bool Perform() = 0;

public: // Non virtual methods
int Result();
const char * Message();
OMNI_Session * Session();
};

I have chosen the Action object as the example (rather than the Session object which is required by all Actions) as it is a class that is extended and thus requires more work to create the bindings:

• every OMNI_Action derived class, needs to only implement the Perform method (to *surprise surprise* perform the method).
• Each OMNI_Action object also needs a OMNI_Session object – the session in which omnifs is running. T his is another class that is exported but not shown.

Step 2: Include necessary Boost headers

Include the required boost headers as follows:

#include <boost/python/module.hpp>
#include <boost/python/def.hpp>
#include <boost/python/class.hpp>
#include <boost/python/args.hpp>
#include <boost/python/overloads.hpp>
#include <boost/python/docstring_options.hpp>
#include <boost/python/enum.hpp>
#include <boost/python/pure_virtual.hpp>

#include <boost/python/return_internal_reference.hpp>
#include <boost/python/copy_const_reference.hpp>
#include <boost/python/copy_non_const_reference.hpp>
#include <boost/python/return_value_policy.hpp>

or alternatively include ALL python related headers by include just python.hpp as:

#include <boosts/python.hpp>

Also enable the use of the python namespace:

using namespace boost::python;

Note that this could be avoided if you prefer to explicitly qualify items with the python namespace each time.

Step 3: Create the module

All the generated python bindings happen inside a module definition. This is done as:

BOOST_PYTHON_MODULE(module_name)

{

// Put binding definitions here – ie class exports and method exports

}

This is it. no need for a main function or anything. Think of the BOOST_PYTHON_MODULE bit as the main entry point of the generated shared object.

Step 4: Export the Classes

As shown in the “Exposing Classes” section of the boost.python tutorial, exporting OMNI_Action is simply a matter of doing:

class_<OMNI_Action>("Action")

.def("Perform",
pure_virtual(&OMNI_ActionAuthenticate::Perform),
"The function that performs the specific action.")
.def("Result", &OMNI_Action::Result,
"Returns the result of the action after \"Perform\" is called.")
.def("Message", &OMNI_Action::Message,
"Returns the message associated with the return result \n"
"after \"Perform\" is called.")
.def("Session", &OMNI_Action::Session, return_internal_reference<>(),
"Reference to the session object on which the action "
"object is valid.")

;

Now there is on SMALL problem with the above. The Perform method is a pure virtual method. Which means that due to a “no implementation” the above class cannot actually be exported. In order to do this, a wrapper class (called ActionWrap) is created for OMNI_Action, and THIS wrapper class is exported instead as shown below:


struct ActionWrap : OMNI_Action, wrapper<OMNI_Action>
{
bool Perform()
{

#if BOOST_WORKAROUND(BOOST_MSVC, <= 1300) // Workaround for vc6/vc7
return call<int>(this->get_override(“Perform”).ptr());
#else
return this->get_override(“Perform”)();
#endif

}

};

This is it. Now we have provided a proxy implementation of the Perform method which essentially calls the derived Perform method when invoked and we have gotten ridden of the pure virtual method.

The exporting of this class is:

class_<ActionWrap, boost::noncopyable>(“Action”,
“A wrapper for classes deriving from the Action object.\n”
“The Action class is the super class of all classes that \n”
“perform work on the omnidrive server.”, no_init)
.def(“Perform”,
pure_virtual(&OMNI_ActionAuthenticate::Perform),
“The function that performs the specific action.”)
.def(“Result”, &OMNI_Action::Result,
“Returns the result of the action after \”Perform\” is called.”)
.def(“Message”, &OMNI_Action::Message,
“Returns the message associated with the return result \n”
“after \”Perform\” is called.”)
.def(“Session”, &OMNI_Action::Session, return_internal_reference<>(),
“Reference to the session object on which the action “
“object is valid.”)
;

Note

• how all the methods of the original OMNI_Action class are also exposed (ie Perform, Result, Message, and Session)
• The noncopyable specifies that the ActionWrap cannot be copied or instantiated
• The “return_internal_reference<>” specifies that the return value of the “Session” object is infact a pointer to a class member, which means that the lifetime of the return value (a session object) is tied to the lifetime of the Action object. So unless the Action object is destroyed, the reference count of the return Session instance cannot be reduced! Please refer to Call Policies in the Boost.python tutorial for more information on this.
• The string values are essentially documentation strings in exported python module (docstrings).
  

Once this is done, the derived classes can be exposed normally.
Step 5: The Build Process

Once you have exposed all the classes in the above fashion, it is time for setting up your builds!

The builds are performed using bjam – the Boost build system. bjam looks for the file Jamroot in the current directory the same way that make looks for a Makefile.

bjam is a lot more complex than make and the bjam tutorial is an excellent source of all the information you are going to need.

I will go through my Jamroot file line by line:

1. Specify the Boost directory: This is the location where boost was installed.

use-project boost
: /opt/boost_1_34_9 ;

2. Specify the requirements and default builds. To make debug the default build, simply specify that in the default-build line.

project
: requirements <library>/boost/python//boost_python
: default-build release
;

3. Specify a dependency on the libomnifs C++ library. The C++ library is infact built in ../bld/<build_mode>

The following two rules mention that in the debug mode the debug build of the libomnifs is to be used and similar in the release mode, the release build of the libomnifs is to be used, as indicated by the “variant” flags.

lib libomnifs
:
: <file><locateion of omnifs source tree>/bld/debug/libomnifs.so <variant>debug
;

lib libomnifs
:
: <file> <omnifs_build_dir_path>/bld/release/libomnifs.so <variant>release
;

4. This specifies the installation target. Unlike in “make” where installations are arbitrary actions, they have a bit more meaning in bjam. These (in this rule) simply specify the installation location (in this case the current directory “.”) and types (install a library instead of an executable):

install dist : omnipy : <location> . <install-type>LIB ;

5. Finally the following specifies that the object being built is a python-extension library (of the name omnipy). Many other types of projects are also possible. Please refer to the bjam tutorial for a full list of build types. The .cc files are the files in which I had divided the export declarations. It is recommended to break up the class export declarations into multiple files so that changes in one class will not require the compilation of the entire source – there by speeding up the build process.

python-extension omnipy
: omnipy.cc omnipy-logger.cc omnipy-utils.cc omnipy-session.cc
omnipy-connection.cc omnipy-memory.cc omnipy-actions.cc libomnifs
;

Step 6 – In Action

Finally to see a glimpse of the library in action, simply start python and run the following:

– Import required modules including the one we just created

>>> import omnipy, sys, os

– create a session object by calling the Session constructor (refer to the actual distribution)
>>> session = omnipy.Session()

– create an actual action object – the action for creating a folder:

>>> action = omnipy.ActionCreateFolder(session, “new_folder1″, “”)

– Perform the action

>>> action.Perform()

and so on and so forth.

Please refer to the example.py and utils.py in the omnifs source distribution for more details.

Step 7: Gotchas and Todos

There a still a few things I havent yet figured out. One of them is how to export FILE * and streams so that we could use Python file handles instead. Once Im done with that Il put that up in here as well.

---

Well that concludes my little tip on how to create python bindings (more of a self-learning set of notes more than anything else). I hope this really saves you a lot of time. Please let me know if you find anything I am missing or forgot or just plainly got wrong. Id be glad to learn from it myself.