This is a writeup of a talk I did recently at Software Passion Summit in Gothenburg, Sweden.

Writing a specification in a full-blown programming language like Python has upsides and downsides. On the downside, Python is not designed as a declarative language, so any attempt to make it declarative (apart from just listing native data types) will require some kind of customization and/or tooling to work. On the upside, having a declaration in the language you write your servers in, you can use the specification itself, rather than a generated derivative of that specification, and writing custom – in this case minimal -generators for other languages is simple, since you can you Python introspection to traverse your specification, and the templating logic of your choice to generate source – this makes it possible, for example, to target a J2ME terminal that just won’t accept existing solutions, and where dropping a 150K jar file for protocol implementation is not an alternative.

For me, this journey started around 2006 when I started to lose control over protocol documentation and protocol versions for the protocol used between terminals and servers in the fleet management solution Visual Units Logistics. After looking for, and discarding, several existing tools, and after being inspired by the fact that we usually configure Javascript in Javascript, I started to sketch (as in, ink on paper) on what a protocol specification in Python would look like. This is a transcription of what I came up with at the time:

imei = long
log_message = string
timestamp = long
voltage = float

log = Message(imei, timestamp,
           log_message, voltage)

protocol = Protocol(log, ...)

protocol.parse(data)

With this as a target, I created the first version of a protocol implementation. It looked similar to the target version, but suffered from an abundance of repetition:

#protocol.py
LOG = 0x023
ALIVE = 0x021

message = Token('message', 'String', 'X')
timestamp = Token('timestamp', 'long', 'q')
signal = Token('signal', 'short', 'h')
voltage = Token('voltage', 'short', 'h')

msg_log = Message('LOG', LOG, timestamp, signal, voltage)
msg_alive = Message('ALIVE', ALIVE, timestamp)

protocol = Protocol(version=1.0, messages=[msg_log,msg_alive])

#usage
from protocol import protocol
parsed_data = protocol.parse(data)
open('Protocol.java’,'w').write(protocol.java_protocol())

The implementation around this is simple; the Token class knows how to parse a part of a message, the Message class knows which Tokens to use (and in which order), and the Protocol class selects the correct Message instance using a mapping of marker bytes to Message instances.

However, no support is given for handling multiple versions of the protocol, and the amount of name duplication makes it really cumbersome – so I set out to create a better version.

Some things complicated the creation of a better version. The worst problem of them all proved to be me, myself and I. At this time I had used Python for a couple of years, and started to get interested in the more sophisticated tools available. I had just taught myself about metaclasses, and thought they were an ingenious application of object orientation – and having found a shiny new hammer, I was itching to find a nail.

Unfortunately, I had no pressing need for using metaclasses, so I invented one – I wanted to avoid some assignments in the protocol specification, so I used metaclasses to rip out the init (constructor) method and replace it with a version that registered the instance in a global map and then called the original init method. This is wrong in at least three ways – since it was not generic, it could have been done in the init method directly, if it would have been general it would have been a job for a decorator, and it is a really great way to obfuscate the code:

__MSG_MAPPING__ = {}

def msg_initizer(cls, old_init):
    def new_init(self, name, marker, *args):
        __MSG_MAPPING__get(cls, {})[name] = self
        __MSG_MAPPING__[cls][struct.pack("!B", marker)] = self
        old_init(self, name, marker, *args)
    return new_init

class RegisterMeta(type):
    def __new__(cls, name, bases, attrs):
        attrs['__init__'] = msg_initizer(cls,
                                         attrs['__init__'])
        return super(RegisterMeta, cls).__new__(cls,
                                      name, bases, attrs)

class Message(object):
    __metaclass__ = RegisterMeta

This is the kind of code I’m not proud of, by the way. The worst part? It didn’t even remove the duplication, although it lowered it somewhat – and the global registration of messages when loading a protocol really messed up any attempt of multiple version support. This was not the only problem; I also went overboard and wanted to support specifying protocol syntax, using a Flow class that defined legal ordering of messages. This might have been a good idea had we actually had any such requirements in our protocols; since they are “authenticate, do anything”, adding support for this just expanded the codebase and made the protocol specification more complex for extremely little gain (especially since we authenticate in different ways depending on the client). Adding insult to injury, this is even more verbose than the very first try.

#In protocol.py
imei = Token('imei', 'long')
message = Token('message', 'String')
timestamp = Token('timestamp', 'long')
signal = Token('signal', 'short')
voltage = Token('voltage', 'short')
auth = Token('auth', 'String')

Markers({'LOG': 0x023,
    'ALIVE': 0x021,
    'AUTH': 0x028})

Message('LOG', imei, timestamp, signal, voltage)
Message('ALIVE', imei, timestamp)
Message('AUTH', imei, timestamp, auth)
Flow([('AUTH'), ('LOG', 'ALIVE')])

#Usage
protocol = Protocol(version=2.0)
parsed_data = protocol.parse(data) #error if not auth parsed

This entire attempt became a warning example – it shows the danger of finding new and interesting technology and applying it before grokking it, and it shows the danger of over engineering and feature creep. Luckily, once I got a good look on what I had created, even me-a-few-years-back could see that this was an abomination, which was subsequently quietly taken out back and put down without even making it as far as integration tests.

Finally, and ongoing, I decided to apply a carefully measured amount of standard library magic to make the specifications more terse, and remove stuff that we did not need. This made the specification look something like this instead:

#In protocol_4.2.py:
#Tokens
t('message', string)
t('timestamp',  i64)
t('signal', i16)
t('voltage', i16)

#Messages
LOG = ('A log message containing some debug info',
     0x023, timestamp, message, signal, voltage)
ALIVE = ('A message to signal that the terminal alive',
     0x021, timestamp)

#Usage
protocols = load_protocols('protocols')
parsed = protocols[4.2].parse(data)
protocols[4.2].write_java() #Writes to Protocol42.java

At one time, it was even terser, but that version didn’t really pan out, and the version in production is very similar to this one. Name duplication is avoided using two different techniques – the tokens are defined by calling a method t that creates the Token instance and injects it back into the calling namespace using the supplied name:

#In types.py
from inspect import currentframe

def t(name, data_type):
    """Inserts name = (name, data_type) in locals()
    of calling scope"""
    currentframe().f_back.f_locals[name] = (name, data_type)

To some, this may seem like blasphemy, but consider this – the implementation is extremely simple in concept, it gets the work done, and it is easy to explain. Another change is that the messages are created solely by using inspect to extract members of the module that look like messages – name in all caps, and a tuple. Worth noting might be that there was error handling initially, but I removed that to make parsing fail, rather than accept a specification that may or may not have contained errors.

Finally, java source and html documentation is created by traversing the protocol instance, and feeding the information into simple templates – experiments were made using literate programming using ReST to create documentation, but in the end that tended to obfuscate rather than the reverse. This may be an effect of naive implementation, or that the problem does not lend itself well to literate programming, but either way it was not worth it in this case.

There is a working and slightly generalized version available at bitbucket, and it you would like to hear more about this (and more details about Python magic used), you can buy a ticket to EuroPython – you’ll have until Sunday to vote for my proposals (and others).