Well back up a bit.

Hands up those who were *forced* to use windows and hence had to install cygwin?

Me and me again!

At work I am on a windoze box while having to ssh into a linux box to do my development (let us not go there).  Luckily cygwin has helped me in the past but only because I managed to complete the installations out of sheer luck.   For some reason (may be limited intelligence) I could not select the fastest or the most reliable mirror and the installations kept bombing out after hours of what seemed like progress.  Seems wierd but I would have thought the cygwin mirror seeing what files it had already downloaded would have helped.  But no such luck (was it some option I missed)?

So I decided to write a downloader myself (you would still have to call the cygwin setup tool to install the sources).  It HAD to be stateful.  You interrupt it at any point and it should continue where it left of.  So out of that came cyginstall.

It also has a few other goodies:

1. Multi threaded (defaults to 5)
2. Web based UI
3. Easily select or deselect mirrors
4. Works through proxy with NTLM authentication.
5. Better reporting of download status and mirror health.

So please give it a go and let me know what you think.  I managed to do a whole download but that does not mean there are no bugs so would appreciate it if you could let me know of any and all issues that need fixing.

Also the UI is very drab.  Very basic but drab, so any help or advice there would be gold.

———————————————–

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads This blog is doing awesome!.

Crunchy numbers

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 4,000 times in 2010. That’s about 10 full 747s.

In 2010, there were 5 new posts, growing the total archive of this blog to 42 posts.

The busiest day of the year was March 31st with 41 views. The most popular post that day was Android Hack – Parcellable in eclipse.

Where did they come from?

The top referring sites in 2010 were anddev.org, rainbowdevil.jp, markmail.org, mail-archive.com, and mindtherobot.com.

Some visitors came searching, mostly for gray code generator, django app engine, google maps captcha, django mobile, and aidl file.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Android Hack – Parcellable in eclipse March 2008
18 comments

Django/Appengine Coexistence, Part 1 – Environment Specific Settings.py February 2010
7 comments

Mango – Mobile Django – Templates March 2010
2 comments

Gray Code Generator October 2009
2 comments

Carnatic Violin March 2007
17 comments

Wish me all luck folks.  Certainly a bit of a pain to be wearing business-y outfit to work every day but hey its small price!

So it hit me.  Why not doctests for C++.  Lo behold DocTest++.  Wrote this last weekend because out of laziness I was’nt really adding test cases to my mango code, as much I should.  Mango certainly does have a few unit tests (and growing rapidly) and even more certainly has no plans to replace those unit tests.  But why not doctests as well?  So DocTest++ extracts tests embedded with C/C++ code and creates test cases for them.  More at the project page on git hub for how it works and what it does.

And to go a step further, this is part of the build process in my make file. A target XXX.test.cpp is (automatically) added for file XXX that contains the @test tag.  This gets compiled along with all tests.  More importantly if any of the files change or new @test tags are added, these are picked by the build system.  Wooot!

Any improvements or feedback?

Anyway, clearly servers on mobile devices instead of server boxes sitting in some rack is exciting.  Imaging you are on the move with a few people on a train and voila you have a webserver you can put up for sharing.  Mind you, this is not that different or even superior to P2P (locally via Bluetooth or globally).  Just another alternative.  So one of the tools I figured Il use quite a bit on my mobile, should it take off as a webserver is a templating engine.  Here is where Mango comes in.  This was actually inspired by Django templates.  And being a big worshipper of Django, I think of Django as a philosophy and not just as a framework.  So first thing was the port of the templating engine.

Where exactly am I porting it to?  First of all the code is opened under an Apache license and is on git.  Secondly I am porting it to android and objective C.  The objective C version is woefully behind, where as the android version has support for a lot of stuff (for, if, include, variables etc) and has a full set of unit tests with it.  Frankly I find android a much much more intuitive platform to develop than XCode (and objective C).  May be it has to do with my familiarity with java.  The objective C version will follow shortly.

So why another templating engine – Well I had envisaged 2 use cases for it:

1. Consume json or xml data from a backend and apply a template on the data from the client (mobile).

2. Mobile generates the content and applies a template to it before sending off the response to a client’s request.

Either way a separate templating engine helps.

Please feel free to try it out. Would appreciate any and all feedback and not to mention help with extending it to a large number of tags and filters.

1. Check it out from git at http://github.com/panyam/mango.

2. Start eclipse (must have java and android)

3. Open the mango and the mangotest projects (from mango/android and mango/android/test respectively).

4. Select the mangotest project and Run As Junit – watch the unit tests.

So anyway, my M.O is to have a single project with the view and controller logic that is common for django and appengine but to have seperate model classes and APIs targetted to each backend.  The models are accessed via the API classes that do all django/appengine specific magic behind the scenes transparent to the views and the controllers.  I have talked more about this in a previous post.  However I did not provide a specific implementation.  This is now open sourced as part of the Djapps project.  At the moment this is still in its infancy but it is a collection of app to do a lot of common tasks like external authentication, provide model accessors transparently based on appengine or django and so on.  Still evolving and I will talk about this in future posts.

But to get started with all this, the first thing that needs to be worked on is settings.py.  For better or worse, this file is where all your app specific settings go.  So what I do is literally have a common_settings.py that i import from settings.py and have environment specific settings.py.  So I include one depending on what kind of environment I am on.  Environments for me would be “appengine dev server”, “appengine production server”, “local django server”, “production djanso server” etc.

So as an example, here is my settings.py.  Very stripped down:

# -*- coding: utf-8 -*-
import os
import djapps

# import all common settings - we will override them later on
from djapps.default_settings import *

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'xxxxx'

# The root folder of the project
PROJ_ROOT           = os.path.abspath(os.path.split(__file__)[0])

TEMPLATE_DIRS = (
 # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
 # Always use forward slashes, even on Windows.
 # Don't forget to use absolute paths, not relative paths.
 PROJ_ROOT + "/templates/"
)

import dev_profiles
# from dev_profiles.profile_dev_gae import *
from dev_profiles.profile_prod_gae import *
# from dev_profiles.profile_dev_nongae import *
# from dev_profiles.profile_prod_nongae import *

if not USING_APPENGINE: ROOT_URLCONF = 'myproject.urls'

Now in the above dev_profiles is the folder where all my different environment based settings files are (I call them profiles).   So an example env specific settings py (profile_prod_gae from above) is:

URL_PATH_PREFIX     = ""
 
USING_APPENGINE     = True
SITE_URL            = "http://myproject.appspot.com"
SITE_NAME           = "myproject"
SERVE_STATIC        = False
USE_TEST_LOGIN      = False
DEBUG               = True
TEMPLATE_DEBUG      = DEBUG
PROJ_STATIC_HOST    = "/static"
DATABASE_ENGINE     = ''
DATABASE_NAME       = 'myproject'
DATABASE_USER       = 'xxx
DATABASE_PASSWORD   = 'yyy'
DATABASE_HOST       = ''
DATABASE_PORT       = ''
FORMAT_PARAM        = "format"
 
EMAIL_HOST          = "localhost"
EMAIL_PORT          = 25
EMAIL_HOST_USER     = ""
EMAIL_HOST_PASSWORD = ""
 
MIDDLEWARE_CLASSES  = (
 'django.middleware.common.CommonMiddleware',
 'djapps.gaeutils.middleware.SessionMiddleware',
 'djapps.auth.local.middleware.LocalAuthMiddleware',
 'djapps.auth.openid.middleware.OpenIDAuthMiddleware',
 # 'djapps.auth.external.middleware.MultiSiteAuthMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.doc.XViewMiddleware',
)
 
TEMPLATE_CONTEXT_PROCESSORS = (
 "djapps.utils.context_processors.site_urls",
 "djapps.utils.context_processors.gae_local_auth",
 "core.context_processors.openid_user",
 'django.core.context_processors.request',
 "django.core.context_processors.debug",
 "django.core.context_processors.i18n",
)
 
INSTALLED_APPS = (
 'djapps.utils',
 'djapps.dynamo',
 'djapps.auth.local',
 'djapps.auth.external',
 'djapps.events',
 'profiles',
)

Thats it.  Nothing escapes customisation this way.  In case y ou are wondering.  The djapps.default_settings.py is just the settings.py that comes when you first create a django project (with startproject).  Atleast it is very close.

Mind you this has served me very very well on past projects and has not required any major rewrites.  One thing that does annoy me is the last line where I have to manually set the settings I have to choose (ie by uncommenting the line).  You can get around this by checking for an environment variable and importing one based on that but again just a personal preference.

Let me know how this goes for you.

“Simple, isnt that just Graycode?” I blurt out.  At which point I was asked to write an algorithm to actually generate all graycodes for a n-bit string.   And this was where I floundered.  Mind you I had just been through 3 and a half hours and writing an algorithm was the last thing on my mind.  It wasnt that I hadnt known how to do it, I just blanked out then then and there – half an hour close to the finish of the interview and I had lost focus.  Stupid stupid me.

Anyway I just got back from a swim and for some reason the demon of the failed interview reared its ugly head and so I figured Id attempt it here (atleast to get some closure, sob sob).  Here goes.

Here are possible gray codes for N = 1, 2, 3 and 4:

Anyway this is only one combination. Any bit could be changed in generating the sequence. The pattern here is that the first half and the second half of the code are “kind of” the same but with the first bit being 0 and 1 respectively and the rest of the bits being reverse of each other.

ie with N = 2,
first half = 00 and 01
second half = 11 and 10

So a simple algorithm would be (in pseudo-haskell):

gc 1, reverse = ["1", "0"] if reverse, else ["0", "1"]
gc N, True =
      ["1" + x for x in gc(N - 1, True) ] ++ ["0" + x for x in gc(N - 1, False) ]
gc N, False =
      ["0" + x for x in gc(N - 1, False) ] ++ ["1" + x for x in gc(N - 1, True) ]

Problem with the above is the storage required in the recursion to store the graycodes for N – 1 (and that too twice). So in python we could avoid this with generators like:

def graycode(numbits, reverse = False):
    if numbits = 1:
        if reverse:
            yield "1"
            yield "0"
        else:
            yield "0"
            yield "1"
    else:
        if reverse:
            # all the "1"s start first
            gcprev = graycode(numbits - 1, True)
            for code in gcprev:
                yield "1" + code

            gcprev = graycode(numbits - 1, False)
            for code in gcprev:
                yield "0" + code
        else:
            # all the "0" start first
            gcprev = graycode(numbits - 1, False)
            for code in gcprev:
                yield "0" + code

            gcprev = graycode(numbits - 1, True)
            for code in gcprev:
                yield "1" + code

and finally an auxilliary function to print out the codes:

def gcgen(numbits, reverse = False):
    for codes in graycode(numbits, reverse):
        print "Code: ", codes

And that was 15 minutes of thinking. Damn it why didnt it come to me last year? Could have been the difference between a googler and foogler!!! Unless ofcourse it didnt really matter and they simply thought I was just a dick@#$D!!!

Django can be quiet easily used with GAE.  And infact that is what I have done.  I use django entirely within GAE.  The real impedence mismatch pops its ugly head thanks to the completely different backend schemes used by GAE and Django (key-value store vs relational DB).  As much as I am not a fan of the GAE, it is a pretty good platform for prototyping.  Well its free and hence, given the state of the competition, the best choice (for now) in the price range.   So how does one use GAE for prototyping an application, leaving it out there to collect feedback (and failing early and failing often) and move the application a platform of choice when the limits are reached and the constraints in GAE are just too much?  A redesign of an app to leverage non-GAE environments is certainly not the way to go.

What I have done is only expose an API from the app point of view.  Hide the back-end specific details behinds its own models.py (django uses models.py to hold the data models for an applications).  Its actually a lot easier than it sounds (if it already sounds easy, then it is even easier).  What is trickier is that one would have to exercise considerable discipline in following this.  Every call in the application should call only the exposed api instead of using the data-backend related calls provided by the platforms (GAE or Django or anything else).  Even leave the implementation and the optimisation to the specific backends.

As an example, let us take the famous “counter” example.  A typical way to have counters in traditional RDBMs based platforms is to have a table that stores the rows of name/counter pairs.  For DBMSes that are strong with write’s this is great.  GAE however recommends a different approach using shards.  Check Brett Slatkin’s video on this.  So how do you choose between these two?

Firstly settings.py needs to have a variable called USING_APPENGINE (or something similar) that will be set to True if the app is running in GAE and False otherwise (ie a native django app running on the test server or on other webservers).

Suppose our counters api is as follows:

def get_counter(counter_name)

def incr_counter(counter_name, byhowmuch = 1)

Now il need 3 files for the api – counters.py, djcounters.py and gaecounters.py.  Only counters.py will be imported in any other part of the application requireing counters.  djcounters.py will have the get and incr counter implementations for traditional RDBMBs and gaecounters.py will have the same for a GAE specific implementation.

counters.py will be:

from django.conf import settings

if settings.USING_APPENGINE:

from gaecounters.py import *

else:

from djcounters.py import *

Thats it.  In your calling app simply do:

import counters

counters.get_counter(“xxx”)

counters.incr_counter(“xxx”, 10)

This is only the beginning.  Clearly this extends to other types of backends as well (ie support TokyoCabinet in the future etc).  When writing an application you shouldnt have to worry about the optimisations.  Nothing stipulates that gaecounters.py will have to implement Brett Slatkin’s stragey when using counters in GAE, but having separate files allows for this optimisation in the future.  Let the api abstract it all away.

Ofcourse it may appear that this will explode the number of functions required in the API.  Not necessarily (ie functions having to bepeated to do filters or object creations or deletions).  Some of the common patterns are object creation, fetching an object by ID or by certain attrib/column values, deleting records.  There are others, but these are the most common ones.

So I have a base djhelpers and gaehelpers file which define object creations, deletions, searches for ANY kind of object class and each one is optimised (sort of) to that specific implementation.

For instance, in django objects can be created with the method:

def create_object(obj_class, **kwds):

    return obj_class.objects.create(**kwds)

while the GAE implementation would be:

def create_object(obj_class, **kwds):

    obj = obj_class(**kwds)

    db.put(obj)

    return obj

Note these are only basic ones.  Other implementations could go through the kwds list and get the parent and id parameters to set these on the object as required.  Alternatively, you could only save optionally and so on.

So an object creation in another part of the app is simply a matter of doing:

create_object(MyObjectClass, a = 1, b = 2, c = 3)

and the object is created without having to worry about the backend type.

This is again a simple way of hiding the implementation details.  The same could be implemented using metaclasses and properties to define a DBObject class whose metaclass would be the implementation specific class for the above and other object specific methods.

The main goal was to have a small group focusing purely on django activity in sydney.  So far so good.  Had our first meetup last tuesday.  Modest turn out and interesting talk by Sam Cavenagh and Shaon Diwakar.  Oh and thanks heaps for Shaon and Sam for actually pushing me to organise the meetup by getting me out of my lazy zone.

And here’s for lots more activity in the sydney django community.

Last weekend was the StartupCampSydney.  A weekend where about 6-10 teams (each with 6-8 members) build, launch and pitch a Startup in one weekend.  Yep one weekend. To be honest I never thought it would work and when asked to be part of this one, I was trying to come up with an excuse to back out of it (before my wife who I hoped would have asked me not to go, urged me to participate…) …  the urging turned out to be a blessing.. that was one of the most intense, enriching, immersed and enlightening product development excercises Ive been in.

The sheer amount of talent from all backgrounds says it all.. and ofcourse meeting and working with so much talent just makes you do things that you never realised were possible.  Our startup is ThreeFeeds.com – a topic/news aggregator. Check it out, we would love your feedback (please forgive us in advance for the bugs).

Check out a full coverage of SCS by Ross Dawson.

Cant wait for the next one.

Point was that with all this catching up we do, we recently (about 6months-1year ago) started using Lua to script a big chunk of the games so that only the core is handled by C++ and everything UI related is palmed off to Lua.  Obvious advantage is that extra builds are not required as compileable source code is not changed and configuration is a lot easier and lot less time consuming.  Again nothing new here.

The problem is that the lua scripts we invoke from C++ have no easy way of debugging them.  So our engineers had been toiling with this by using the first-class technique of peppering the code with print statements.  You would thing some methodologies would die eventually but nope.

So I figured why not simply have a debugger that takes advantage of Lua’s debug api and allow debugging of scripts remotely.  So was born LunarProbe.  My attemt at debugging lua scripts once embedded in a host language.  Even at the beginning I figured that this was a very generic thing and has no specific code for our games.  So I decided to release it on the Apache License.  You can find the code on google code.

So far it only supports C/C++ but hope to have more languages supported with the appropriate bindings in place.  The idea is simple:

1. The lua_State object (ie the lua stack or the lua context) is god in Lua.  Essentially it is what the interpreter uses to execute scripts and lua statements “on”.
2. A new lua stack can be opened with the lua_open command.
3. Usually following this command, you open all the standard libraries with the luaL_openlibs.

That is it.  You do usual things like loading lua files on a particular stack.  See the lua manual for more info on this.

So where does the debugging come in?  Well each lua stack can be given a debug hook function that is called by the interpreter after certain points at runtime. eg after a line has been executed, after a function is invoked, after a function has returned etc.

What LunarProbe does is registers itself as a hook function and deals with each of the above debug events and presents it via tcp (port 9999 by default) to a remote client.  The debug server is in fact api driven.  It accepts commands from the client and returns responses (but not HTTP – to keep it simple).  This means any UI could be written easily by simply following the debug server protocol.  A reference client is provided (written in python).  The python client is supposed to be like gdb (but no where near it!! GDB – please accept my apologies for even trying a comparison!!  I know I dont deserve it.)

I wanted to write a GUI or an Eclipse plugin, but we dont use eclipse plugin at work and wx widget toolkits werent installed by default here at work.  So doing either meant having all developers install extra packages which was a huge adminstrative overhead.  At a later stage, I would like to see more coming up.  Any helpers?

Again I digress.  So how would one enable LunarProbe in stacks they are opening?  Simple.

Instead of lua_open and luaL_openlibs, simply do this:

1. Include the lunarprobe header (ensure that lunarprobe/src is in the include path):

• #include “lpmain.h”

2. Instead of lua_open and luaL_openlibs simply do:

LUNARPROBE_NS::LuaUtils::NewLuaStack(true, true, “hello”)

This does a few things:

1. Creates a lua stack using lua_open
2. If the first parameter is true, then Opens standard libraries using luaL_openlibs.
3. and if the second parameter is true, registers LunarProbe by calling LUNARPROBE_NS::LunarProbe::Attach(stack, name), where “stack” was opened in step 1.
4. The fourth parameter (defaulting to “”) only gives a name to each of your stacks for debugging purposes.  This is very useful in the debug client to identify which stack your dealing with when the address of a stack may not be very useful or very clear.

Alternatively, you can simply call the step 3 instead of NewLuaStack if you do not want to pull out your own versions of lua_open and luaL_openlibs.

That it. This is the embedding/attaching part of the debugger.  Now the client.

The Client

lunarprobe/client includes a simple client – client.py.

I wont go into too much detail here as the “help” command is mostly self-explanatory.  Just the brief stuff.

Invoke this with:

• python client run

or if you would like to run it from within an interpreter then:

• import client
• client.run()

This brings up the LDB prompt: type in “help” to show all commands and their usage.

One distinction (actually an inconvinience) is that since multiple stacks can be debugged at the same time, any context (LP calls stacks as contexts) specific commands must pass in the address of the context.  This is not hard, just inconvinient.

To get a list of all the contexts simply type in the contexts at the prompt and press enter.  This will show all context names and their addresses.  Use these addresses with any context related commands (like step, next, continue etc).

Breakpoints at the moment can be set by filename (and linenumber) or by a function name.  However at this stage member functions are not differentiated from global functions.  So a function name breakpoint will affect ALL functions of the same name.

As mentioned this is very brief.  Apologies for that.  Il add more and more in as I improve this.

Well hopefully this project is useful for all Lua embedders out there.  Please let me know how I can improve this.  Also this project is on google code, so please let me know if you would like to contribute or extend LP.

Best of luck!!!

August – Had my exam for Marketing Management and Stats.  For some reason i find that more I learn in a subject the worse I do in the exams.  I really need to learn to write academically!  (Yes I know it sounds like a looser and that most likely must be it!)

September – 3 weeks in India.  Wow Wow Wow!  Entrepreneurship is just springing up.. (ok the financial climate is not so great but that is really an issue of the system rather than the product/environment)… After “the” valley India is the place to be.  Actually not entirely true.  Even though many wouldnt agree with me, Id say china is still leading in that aspect as people are a lot more risk taking than in India but hey Ive never been to china so it is a very very brave and untested statement.

October – 2 weeks in HongKong for one of my subjects (Private Equity and Venture Capital)..  amazing course.. only sad thing was that I had to go all the way to HK to do a course in entrepreneurship, which was not offered by AGSM in sydney… what does that really say about the apetite for entrepreneurship in australia?  But loved HK – the city and the people… I cant say the same about the food.  See I am a vegetarian and this is something not easily catered.  Also I was a bit dissapointed with the prices.  Not exactly as cheap as I hoped them to be.  Oh well..

November – Had my exam for the PE and VC subject..  confirms my theory about learning being inversely proportional to my mark… well havent got my mark yet but not holding my breath for a great mark… damn

December – Had 2 concerts yesterday for SMC.  Tiring but lots of fun.  One more in Brisbane on the 13th and one possibly in melbourne during christmas.  Have to see.  Both very challenging and testing.  One at a time.

Ciao till later.  Will be updating on a lot of programming stuff Ive been hacking away on.

The course is very insightful… if you follow VC news on techcrunch and a whole bunch of blogs from the big guns in the Valley (no i am just a follower not a propogator), this course is very very complementary.  Ofcourse it is quite sad that we (students) in the Australian campus of AGSM have to resort to travelling overseas to undertake a course in anything entrepreneurial.  Seems the Australian focus is on management training more than anything else.  Could that explain the pitiable state of entrepreneurship in Australia?

Apple’s reaction to the app is hardly surprising, after all the company does have a reputation for masking its dubious actions of cencorship and spoon feeding its customers under the blanket motto of “quality”.  What was surprising (may be less so given the psyche of the iStuff customers) is the customers reaction.  All I have heard all around are “kudos Steve, great work.”

Forget cencorship, are we actually planning to get rid of the words “Common Sense” out of our vocabulary and lives for ever?  What ever happened to Buyer Beware?  Let me go through the arguments Ive heard.

1. It is Apple’s store and they can do what ever they want.

Ok No argument here.  I mean if M$ can do what they want and get away with it, why shouldnt Apple?  If this is all we care about, then stop reading right here.

2. You dont find $1000 news papers in a news-stand so why should we have them in the AppStore?

Marketing and Economics 101.  A news agent, has to prioritise the shelf space to maximise his/her profits.  A good indication of this is the customer demand for a product, not ethical or moral values.  You dont find $1000 news papers in a news stand simply because the market segment looking for $1000 news papers is not the one being targetted by the news stand.  If you are looking for expensive magazines try this.

3. Dilutes the quality of the App Store

Welcome to economics again.  Ever heard of supply and demand?  Ever heard of transaction costs? It was not exactly hard to “find” the product.  The App store enables you to search by price and (I am assuming) a few other criteria.  So we have 0 transactions costs for the users OR Apple (never mind that Apple made a cool 30% of the app!).  In an open market (which is what this SHOULD be) the best would bubble to the top and crap like this sinks to the bottom (ok I suddenly have a hankering for this app and I dont even have an iPhone!).

4. People are being fooled and tricked.

How?  Was the price not clear?  Was the product description not clear?  Sure the app is useless.  But was its uselessness ever a secret and mis-communicated?  The 8 who have actually bought the app, knew exactly what they were getting into (by the way how many of them are actually complaining about it?).

5. Mockery of the store

Wow another insightful argument for masking the envy at good (well ok going a bit far there) entrepreneurship.  So suddenly we are ok with Louis Vuitton bags (known mainly for their durability?) but an attempt at brand management and positioning (towards deep-pocketed) users is wrong and unethical and evil and dodgy (and a whole lot of other equally _____ adjectives).  Perhaps it is time we (developers) start making apps targeted at user needs (refer to PG’s “make something users want” mantra for this)?

————————

All said and done fellas, the marketing hype for the AppStore and Heinrich (the app’s author) have been phenominal.  Even with a sale to 8 customers, thats a massive revenue of 80,000.  Wow.  Kudos.  This is just capitalism guys.  Get used to it.  The thing with idiots is, if you cant beat them, you just take advantage of them.  And kudos again to Heinrich for doing just that.  Dont try to beat this guy, join him!!

The other reason I am excited about Apple having been foolish in dealing with this App, is that the need for a better market place for the iPhone is becoming more pronounced.  How long can Apple’s cencorship and treatment of its customers as children help insulate (and incubate?) its customers’ stupidity?

So as a simple project I figured Il write (yet another online) chess game – WizChess.  Note that there is already a Blitz Chess as part of the appengine samples page.  I wanted to write one just to get a feel for the engine and to largely to learn a lot of things myself.  There are a still a few things I have not implemented (eg on a castle, the server updates, but the client does not properly update UI.  en-passante-s are not implemented.  Timed game not hapening yet, chatting between players not yet happening and so on).

It was a very interesting experience.  The documentation was quite nice and full, but still I was not fully satisfied.  For some reason I found django a lot more satisfying as a web-app dev platform.  For the models for the game I had chosen to follow a more django style (relational) rather than harnessing the rich model extensions provided by appengine (mainly the Expando Models).  I had done this as I wasnt sure if I wanted to seal my app purely to the AppEngine platform.  And after finishing the project, I am still not sure that I do.

I had got a whole bunch of puzzle games from here.  Now I had needed a way to update the database with an initial data set.  So my way was to have a url on the webapp (something like /resetdb) that would load the initial board templates (and in the future games that are saved and so on).  However, since this loading would take some time, I had to resort to breaking this up into several parts and calling each of these individually.  eg:  Id nomally have:

class ResetDB(webapp.RequestHandler):
    def get(self):
        for board in my_boards:
            add_board_to_db(board)

Now as the number boards in my data set increased, the time to put them in the data store would naturally increase.  But there seems to a time limit on each request, which would make my request fail.  So I had to resort to writing a client side script that would call ResetDB with parameters (eg, firstboard and lastboard) and then also break em up as more saved games were restored etc.  Absolute Pain!  Its all good for appengine to be smart about things but to impose assumptions severely limits the usefulness of this platform and I hope in the future this is released.

Next thing was the model apis.  The Expando model is awesome, but once again I was not happy about the lockin.  But this was not my main concern.  The querying api is extremely poor.  Django has amazing querying facilities where one can specify pretty much arbitrary query filters (not a and c or a minus x and so on).  AppEngine does not support “not”s in its query.  Aaaaaaaaaaaaaaaaah.   Also I couldnt find an easy way of chaining queries.  Eg, to find a game, where either player0 or player1 (my way of saying white or black) was user, I had to do this:

games_with_user = []
games = models.Game.all()
games_with_user.extend([g for g in games if g['player0'] = current_user])
games_with_user.extend([g for g in games if g['player1'] = current_user])

Yuck!  Same thing in Django happens with:

games_with_user = models.Game.filter(Q(player0 = current_user) or Q(player1 = current_user))

(ok I did that syntax from memory so it may not be right, but it is something similar).

Now the final that REALLY annoyed me was lack of Comet (or any reverse ajax) support.  Essentially I poll the server every 5 seconds to get updates on whether the opponent has played.  Would have been great for the server to notify the client instead of having to keep polling and waste requests (I believe there is a limit of 6.5 million requests a day or is that 650K requests?)

But having said all this, the main (only?) advantage of the platform is the fact that we dont have to worry about scalability.  Unfortunately this is a HUGE advantage.  Normally I could write an app, and host it on my own boxes at home (if i want it to be free), but then il soon be hitting limits with 10 users and Il have to find a hoster and also handle scalability issues myself.  So migrating my own code to appengine at THAT point would be quite a nightmare.  By writing and prototyping it on appengine allows me to fail fast.  Looking at the pricing, app engine would charge about $40 per month per app (with reasonable traffic).  If my app ever got to that stage, I could always port it out of appengine (as long as I dont use Expando and stick to django style object modelling – with which I dont loose any power).

Please let me know if you would like the sources.  I am not sure how to upload files to wordpress here.

Also I would really appreciate it if you could give me feedback on my learning here.  As a python and web-app noob, Id be grateful for pointing any errors in my ways

I have had the extreme good fortune of being pointed to a few clips on Youtube by Akella Mallikarjuna Sharma.  Here they are:

http://youtube.com/results?search_query=msakella

Hope you benefit from these.  Good luck.

What I find surprising is will there be any input from Marc on FB’s technical strategy?  I am specifically concerned about Marc’s Ning using OpenSocial and FB using its own FB API.  Will we see a merging of the two?  Surely it would be stupid (a quality I possess in abundance) to think FB would abandon FB API in favour of Ning for better tapping into Ning and its countless social networks hosted within.  However one cannot help see the allure in a wrapper to FB API to be OpenSocial compatible?

Or is this issue too puny and insignificant to deserve a mention?

It is pretty easy (or comforting atleast) sitting in your parents’ basement coding or being sponsored to spend summers on startup ideas sponsored by THE man PG himself (an opportunity Id kill for), but leaving all you have with very little promise of great rewards, purely for the sake of your family, is what I consider true entrepreneurship (well children are the ultimate startups anyway).  Costs of failures are very high (hardship for entire family, loosing face, etc).  Rewards are not staggering except that life for one’s family is slightly better.

I would like to thank two such migrants who risked very comfortable jobs for the sake of their two sons.  Myself and my brother.  Mum was a very respected high-school teacher (she was my math teacher – which could explain my interest in maths).  Dad was a successful accountant.  Like others they had left it all for us.  The main reason was reverse-discrimination in the society gripped by the plague of the caste system, existing only to empower already corrupt politicians (redundant adjective). Times were hard for the first two years – time they had spent unemployed, living on government assistance.  But I am glad for those times.  It had instilled in us a passion and respect for hardwork.  Cant complain at all.  No room for becoming spoilt brats.  In fact I wonder if I can ever replicate their own dedication and values on my own kids (one day).  Atleast I know that I dont have to look far in search of role-models!

So thanks mum and dad (hope this combined post suffices for mothers’ and fathers’ days).

I was a bit skeptical to see how it actually works but man its amazing!  Looks like I wont have to be working ever again!!

Here’s the link!!

http://www.google.com.au/intl/en/gday/index.html

wow. Amazing platform. Ok so the sdk is not all that well documented but it is getting better by the day. Ofcourse due to its young age, it is nowhere near the flex sdk in terms of the ease of use, documentation and ide, but hey a great start. And there are tons of samples out there ofcourse.

Anyhoo today is about one of the many problems and a fix I stumbled across – Including Parcelables in eclipse projects.

Background:

It is very easy to create remote services in android. A remote service runs in the corner handling requests by activities (or other processes in general) that need to use them (essentially server like processes). By following the tutorial above, you will find that “aidl” files come in 2 flavours.

1. Include the interface definition of an interface.

2. Contain a single statement “parcelable xxxx”, indicating that class xxxx can be marshalled and unmarshalled (into Parcel objects) so instances of xxxx can be communicated between processes.

With step 1, an aidl file YYYYY.aidl automatically gets built into YYYYY.java which contains a lot of stub and other helper functions to take care marshalling and remote method invocation and yada yada yada. This seems to be an automatic thing with eclipse.

With step 2, the .java already exists, and hence the need to use the “parcelable” keyword to indicate that “there exists a corresponding XXXX.java that is already parcelable so do NOT create a .java file again”, but eclipse does not understand this.

Worse still, it is intuitive to place the aidl files along with all your other sources. The fix involves NOT doing this.

Your project will have usually have the following structure:

<project>

—- .classpath

—- .project

—- .settings/

—- bin/

—- src/

——– package1

————file1.java —> This is parcellable

————file2.java

——– package2

————interface1.java

————class2.java —-> This is parcellable

and so on

the key is to add another folder at the same level and structure as the “src” folder(I called mine aidl).

So the new structure looks like this:

<project>

—- .classpath

—- .project

—- .settings/

—- bin/

—- src/

——– package1

————file1.java —> This is parcellable
————file2.java

——– package2

————interface1.java

————class2.java —-> This is parcellable

—- aidl/

——– package1

————file1.aidl —> This is parcellable
——– package2

———— class1.aidl —-> a sample interface definition with no corresponding .java files ANYWHERE

————class2.aidl —-> This is parcellable
the aidl files that exist in your src folder, have their corresponding aidl files in the aidl folder in the same package structure.

Now good news is since the aidl folder exists in your <project> folder, it will be picked up by eclipse (on a refresh). However there is still one minor issue. aidl file class1.aidl will not be automatically compiled to class1.java. To do this, we need to add the entire aidl folder as a “source” folder. Simple. do this:

add an entry like so in the .classpath file!! (ah ha)

<classpathentry kind=”src” path=”aidl”/>

This essentially says, “i have another folder called ‘aidl‘ that is a source folder”.. voila, being a source folder it will get included in auto-builds.

Beauty of this is that for class2.aidl and file1.aidl (in your aidl folder), no .java file will be created as they already exist, however class1.java will be built.

Saves a huge pain in the rear side (i couldnt spell “posterior”)!

good luck. please let me know if ive missed something.

PS <08/03/2008>

Please see Comment # 2 for further clarifications (and speculations too).  Thanks for trickybit (Comment #1) for inspring that!

Hate captcha in the google maps.  I can understand how it needs to protect itself against spam, but this is ludicrous.  I was playing with the google maps api writing a simple google maps mashup.  This meant editing my js file and reloading the pages a few times.  Somehow google flagged this as a sequence of spam requests.  So it shows me the dreaded captcha page.  I enter the damn captcha.  You would think a company like google would be smart enough to let you get on with your work.  Oh no, it brings up the captcha page again.  May be I got the captcha wrong originally so I tried again, no luck.  Now onto my 100th try in a row and still no luck.  Google still thinks my requests (from a FF3 browser) are all spam requests.

Is this the sign that gOoGle is finally going the MS way?

Essentially data is fed into the parser continuosly (as there are no blocked reads in flex from the socket).

Usage is:


var parser:GPJsonParser = new GPJsonParser()
parser.processBytes(byteArray)
var nextObj:* = parser.next();
while (nextObj != null)
{
// do something with nextObj
nextObj = parser.next();
}


nextObj will only be non-null if there were no syntax errors and if enough bytes were consumed to produced a complete json fragment.

package gprobe.utils
{


import flash.utils.ByteArray;/*
* On-demand parser for parsing json from a stream
*/
public class GPJsonParser
{
private static var STATE_EOF:int                  = -1;
private static var STATE_START:int                = 0;
private static var STATE_READING_IDENTIFIER:int    = 1;
private static var STATE_READING_NUMBER:int       = 2;
private static var STATE_READING_STRING:int       = 3;

private static var OBJ_NULL:int         = 0;
private static var OBJ_BOOL:int         = 1;
private static var OBJ_NUMBER:int       = 2;
private static var OBJ_STRING:int       = 3;
private static var OBJ_LIST:int         = 4;
private static var OBJ_STRUCT:int       = 5;
private static var OBJ_COMA:int         = 6;
private static var OBJ_COLON:int        = 7;
private static var OBJ_OPEN_BRACE:int   = 8;
private static var OBJ_OPEN_SQUARE:int  = 9;
private static var OBJ_CLOSE_BRACE:int  = 10;
private static var OBJ_CLOSE_SQUARE:int = 11;

/*
* Current string delimiter only valid if we are
* currently reading a string
*/
private var currDelim:int;

/*
* Current string - only if we are reading a string
*/
private var currString:String;

/*
* State in reading current string
*/
private var currStringState:int;

/*
* The byte buffer
*/
private var byteBuffers: Array = new Array();

/*
*
*/
private var currBuffer: ByteArray;

/*
* Number of bytes read since start of object
*/
private var bytesRead: int;

/*
* Current parser state
*/
private var currState: int;

/*
* How many unmatched open square brackets have we encountered?
*/
private var squareCount: int = 0;

/*
* How many unmatched open braces have we encountered?
*/
private var braceCount: int = 0;

/*
* Current parent object
*/
private var objStack: Array = null;

private static const CHAR_BSLASH:int = '\/'.charCodeAt(0);
private static const CHAR_DQUOTE:int = '"'.charCodeAt(0);
private static const CHAR_SQUOTE:int = '\''.charCodeAt(0);
private static const CHAR_PLUS:int = '+'.charCodeAt(0);
private static const CHAR_DOT:int = '.'.charCodeAt(0);
private static const CHAR_a:int = 'a'.charCodeAt(0);
private static const CHAR_A:int = 'A'.charCodeAt(0);
private static const CHAR_e:int = 'e'.charCodeAt(0);
private static const CHAR_E:int = 'E'.charCodeAt(0);
private static const CHAR_t:int = 't'.charCodeAt(0);
private static const CHAR_v:int = 'v'.charCodeAt(0);
private static const CHAR_b:int = 'b'.charCodeAt(0);
private static const CHAR_f:int = 'f'.charCodeAt(0);
private static const CHAR_r:int = 'r'.charCodeAt(0);
private static const CHAR_n:int = 'n'.charCodeAt(0);
private static const CHAR_z:int = 'z'.charCodeAt(0);
private static const CHAR_Z:int = 'Z'.charCodeAt(0);
private static const CHAR_0:int = '0'.charCodeAt(0);
private static const CHAR_9:int = '9'.charCodeAt(0);
private static const CHAR_SLASH:int = '\\'.charCodeAt(0);
private static const CHAR_SPACE:int = ' '.charCodeAt(0);
private static const CHAR_MINUS:int = '-'.charCodeAt(0);
private static const CHAR_UND:int = '_'.charCodeAt(0);
private static const CHAR_TAB:int = '\t'.charCodeAt(0);
private static const CHAR_CR:int = '\r'.charCodeAt(0);
private static const CHAR_LF:int = '\n'.charCodeAt(0);
private static const CHAR_COMA:int = ','.charCodeAt(0);
private static const CHAR_COLON:int = ':'.charCodeAt(0);
private static const CHAR_OSQ:int = '['.charCodeAt(0);
private static const CHAR_CSQ:int = ']'.charCodeAt(0);
private static const CHAR_OBRACE:int = '{'.charCodeAt(0);
private static const CHAR_CBRACE:int = '}'.charCodeAt(0);

/*
* Constructor
*/
public function GPJsonParser()
{
reset();
}

/*
* Resets the parser
*/
public function reset(resetBytes:Boolean = true): void
{
objStack        = [];
currState       = STATE_START;
bytesRead       = 0;
currString      = "";
currStringState = 0;
squareCount     = 0;
braceCount      = 0;
if (resetBytes)
{
byteBuffers = new Array();
currBuffer  = null;
}
}

/*
* Process more bytes.
*
* Returns nothing.  Call next() to retrieve the next object
* from the stream (if any).
*/
public function processBytes(data:ByteArray): void
{
byteBuffers.push(data);
}

/*
* Return the next object from the stream.
* null if none (this is not necessarily an error, just means we do
* not have enough bytes given to us to extract the next object).
*/
public function next(): *
{
var currCh:*;
var readChar:Boolean = true;

while ( ! (objStack.length == 1 && squareCount == 0 && braceCount == 0))
{
if (currBuffer == null || currBuffer.bytesAvailable == 0)
{
if (byteBuffers.length == 0)
{
return null;
}
currBuffer = byteBuffers.pop();
}
if (readChar)
{
if (currBuffer.bytesAvailable == 0)
return null;
bytesRead++;
currCh = currBuffer.readByte();
}
else
{
readChar = true;
}

if (currState == STATE_READING_STRING)
{
processStringBytes(currCh);
}
else if (currState == STATE_READING_IDENTIFIER)
{
if ((currCh >= CHAR_a && currCh <= CHAR_z)         ||
(currCh >= CHAR_A && currCh <= CHAR_Z)     ||
(currCh >= CHAR_0 && currCh <= CHAR_9)     ||
currCh == CHAR_UND)
{
currString += String.fromCharCode(currCh);
}
else
{
if (currString == "true")
{
shift(OBJ_BOOL, true);
}
else if (currString == "false")
{
shift(OBJ_BOOL, false);
}
else if (currString == "null")
{
shift(OBJ_NULL, null);
}
else
{
shift(OBJ_STRING, currString);
//throw new Error("Invalid identifier: " + currString);
}

currState = STATE_START;
readChar = false;
}
}
else if (currState == STATE_READING_NUMBER)
{
if ( ! processNumberBytes(currCh))
{
readChar = false;
}
}
else if (currState == STATE_START)
{
if (currCh == CHAR_SPACE || currCh == CHAR_TAB ||
currCh == CHAR_CR || currCh == CHAR_LF)
{
// ignore white spaces
}
else if ((currCh >= CHAR_a && currCh <= CHAR_z)         ||
(currCh >= CHAR_A && currCh <= CHAR_Z)     ||
currCh == CHAR_UND)
{
currState = STATE_READING_IDENTIFIER;
currString = String.fromCharCode(currCh);
}
else if (currCh == CHAR_MINUS || (currCh >= CHAR_0 && currCh <= CHAR_9))
{
currState = STATE_READING_NUMBER;
currStringState = 0;
processNumberBytes(currCh);
}
else if (currCh == CHAR_SQUOTE || currCh == CHAR_DQUOTE)
{
currDelim = currCh;
currString = "";
currState = STATE_READING_STRING;
}
else if (currCh == CHAR_COMA)
{
shift(OBJ_COMA);
}
else if (currCh == CHAR_COLON)
{
shift(OBJ_COLON);
}
else if (currCh == CHAR_OSQ)
{
squareCount++;
shift(OBJ_OPEN_SQUARE);
}
else if (currCh == CHAR_OBRACE)
{
braceCount++;
// shift(OBJ_STRUCT, new Object());
shift(OBJ_OPEN_BRACE);
}
else if (currCh == CHAR_CSQ)
{
squareCount--;
shift(OBJ_CLOSE_SQUARE);
}
else if (currCh == CHAR_CBRACE)
{
braceCount--;
shift(OBJ_CLOSE_BRACE);
}
}
}

if (objStack.length == 1)
{
var obj:Object = objStack.pop();
reset(false);
return obj['value'];
}

return null;
}

/*
* Shifts a token onto the stack and immediately reduces it (the
* stack)
*/
private function shift(tok:int, value: * = null): void
{
if (tok == OBJ_CLOSE_SQUARE || tok == OBJ_CLOSE_BRACE)
{
reduce(tok);
}
else
{
objStack.push({'type': tok, 'value': value});
}
}

/*
* Reduces the parse stackby utilising the next token.
* Reduction can happen more than once.
*/
private function reduce(topType:int): void
{
var t1: *;
var t2: *;

if (topType == OBJ_CLOSE_SQUARE)
{
// we are building a list
var newArray: Array = new Array();
while (objStack.length > 0 &&
objStack[objStack.length - 1]['type'] != OBJ_OPEN_SQUARE)
{
t1 = objStack.pop();
if (t1['type'] == OBJ_COMA)
t1 = objStack.pop();
newArray.push(t1['value']);
}
if (objStack.length == 0 || objStack[objStack.length - 1]['type'] != OBJ_OPEN_SQUARE)
{
throw new Error("Could not find '['");
}
objStack.pop();
newArray.reverse();
objStack.push({'type': OBJ_LIST, 'value': newArray});
}
else
{
// we are building a map
var newObj: Object = new Object();
while (objStack.length > 0 &&
objStack[objStack.length - 1]['type'] != OBJ_OPEN_BRACE)
{
t1 = objStack.pop();
if (t1['type'] == OBJ_COMA)
{
t1 = objStack.pop();
}

t2 = objStack.pop();
if (t2['type'] != OBJ_COLON)
{
throw new Error("':' expected but not found");
}
t2 = objStack.pop();

newObj[t2['value']] = t1['value'];
}
if (objStack.length == 0 || objStack[objStack.length - 1]['type'] != OBJ_OPEN_BRACE)
{
throw new Error("Could not find '{'");
}
objStack.pop();
objStack.push({'type': OBJ_STRUCT, 'value': newObj});
}
}

private function processNumberBytes(currCh: *): Boolean
{
var NUM_START:int               = 0;
var NUM_READING_INT:int         = 1;
var NUM_READING_DOT:int         = 2;
var NUM_READING_FRAC:int        = 3;
var NUM_READING_EXP:int         = 4;
var NUM_READING_EXP_DIGITS:int  = 5;

if (currStringState == NUM_START)
{
currString         = "";
currStringState = NUM_READING_INT;
if (currCh == CHAR_MINUS)
{
currString     = '-';
return true;
}
}

if (currStringState == NUM_READING_INT)
{
if (currCh >= CHAR_0 && currCh <= CHAR_9)
{
currString += String.fromCharCode(currCh);
return true;
}
else
{
currStringState = NUM_READING_DOT;
}
}

if (currStringState == NUM_READING_DOT)
{
if (currCh == CHAR_DOT)
{
currString += '.';
return true;
}
else if (currCh == CHAR_e || currCh == CHAR_E)
{
currString += 'e';
currStringState = NUM_READING_EXP;
return true;
}

currStringState = NUM_READING_FRAC;
}

if (currStringState == NUM_READING_FRAC)
{
if (currCh >= CHAR_0 && currCh <= CHAR_9)
{
currString += String.fromCharCode(currCh);
return true;
}
else if (currCh == CHAR_e || currCh == CHAR_E)
{
currString += 'e';
currStringState = NUM_READING_EXP;
return true;
}
}

if (currStringState == NUM_READING_EXP)
{
currStringState = NUM_READING_EXP_DIGITS;
if (currCh == CHAR_PLUS)
{
currString += '+';
return true;
}
else if (currCh == CHAR_MINUS)
{
currString += '-';
return true;
}
}

if (currStringState == NUM_READING_EXP_DIGITS)
{
if (currCh >= CHAR_0 && currCh <= CHAR_9)
{
currString += String.fromCharCode(currCh);
return true;
}
}

currState = STATE_START;
shift(OBJ_NUMBER, Number(currString));

// invalid char, must be end of numeric
// value so let the caller take care of the
// rest by not consuming the character
return false;
}
private function processStringBytes(currCh: *): void
{
var STR_NORMAL:int = 0;
var STR_SLASH:int  = 1;

if (currStringState == STR_SLASH)
{
currStringState = STR_NORMAL
if (currCh == CHAR_SLASH)
{
currString += '\\';
}
else if (currCh == CHAR_t)
{
currString += '\t';
}
else if (currCh == CHAR_n)
{
currString += '\n';
}
else if (currCh == CHAR_r)
{
currString += '\r';
}
else if (currCh == CHAR_b)
{
currString += '\b';
}
else if (currCh == CHAR_f)
{
currString += '\f';
}
else if (currCh == CHAR_BSLASH)
{
currString += '\/';
}
else if (currCh == CHAR_DQUOTE)
{
currString += '"';
}
else if (currCh == CHAR_SQUOTE)
{
currString += '\'';
}
else
{
currString += ("\\" + currCh);
}
}
else if (currCh == currDelim)
{
currState = STATE_START;
shift(OBJ_STRING, currString);
}
else
{
currString += String.fromCharCode(currCh);
}
}
}

}
Essentially, since data is read asynchronously, it needs to be "fed" data as it is recieved by calling

Now the problem was that I could not find a way of debugging the decoding (on the flex side) of the AMF stream I had generated from my server.  Why do I need debugging you ask?  Well, I was having random errors in the AMF stream (no doubt due to my encoding) while encoding strings.  Due to time-constraints I had decided to switch to JSON.  This meant I had to write a json parser for flex (as I could not find one at the time).  Not too bad as it gave me a lot of insight into what was being sent and so on.  One day when I earn more free time at work I may decided to debug my AMF encoder.

Been a great vacation.  Drove down to Melbourne to visit the parents.  Boy that was relaxing and tiring at the same time.  Cant believe time just dissapeared before I knew it!

Any hoo a happy 2008 to all.  Hope 2007 was a great year for all of you and hope 2008 is much better!