Noah Kantrowitz

LazyProfile middleware

2011-02-20T00:00:00-08:00

The topic of extending the User model in Django came up recently on Convore. In the discussion I mentioned a minor hack we had come up with at work to reduce the friction of working with user profile models (which are still the safest way to add extra fields to User).

The LazyProfileMiddleware adds a request.profile in the same style as AuthenticationMiddleware adds a request.user. The only point of warning is that also like AuthenticationMiddleware, this cannot be enabled for only one site in a multi-tenant deployment scenario.

class LazyProfileMiddleware(object):
    """Middleware to attach a lazy .profile value to all requests."""

    lazy_profile = property(lambda self: self.user.get_profile() if self.user else None)

    def process_request(self, request):
        request.__class__.profile = self.lazy_profile

Using SELECT FOR UPDATE in Django

2011-01-16T00:00:00-08:00

A frequently overlooked feature of SQL is the FOR UPDATE clause on SELECT. Even in this age of NoSQL datastores, most large-scale webapps will use a relational database somewhere, and good money would bet on most having a transaction race condition somewhere.

Transaction isolation in a nutshell

Transaction isolation is the mechanism by which a SQL database prevents multiple in-progress transactions from interfering with each other. Postgres provides two different levels, read committed and serializable. Read committed is the default and is sufficient for the vast majority of operations. A simple mental model for how this works is to have a dirty flag on each row. Any time a row is touched by an UPDATE or DELETE statement, the dirty is set. Similarly any time a SELECT is about to operate on a dirty row, more specifically one which was marked dirty by another transaction, it will block and wait for the other transaction to either commit or rollback and will then update the value it was trying to access.

Using UPDATE

As long as you can express all your operations using SQL expressions in an UPDATE, this isolation is sufficient. For example, you can increment an integer column via UPDATE foo SET x=x+1 WHERE id=1;. Unfortunately Django doesn’t make this easy. The naive way to perform the same operation is:

obj = Foo.objects.get(id=1)
obj.x += 1
obj.save()

This means that two requests can come in simultaneously and the second will clobber the first:

A: obj = Foo.objects.get(id=1) # obj.x is now 1
B: obj = Foo.objects.get(id=1)
A: obj.x += 1 # obj.x is now 2
B: obj.x += 1 
A: obj.save() # obj.x saved as 2, as expected
B: obj.save() # obj.x is still only 2, when it should be 3

To do this correctly in plain Django calls, we can use QuerySet.update and an F expression:

Foo.objects.filter(id=1).update(x=F('x')+1)

This works, but the syntax is a bit unfortunate, even moreso when you just want to update a field on a model you already have. I highly recommend using Andy McCurdy’s update method:

obj = Foo.objects.get(id=1)
update(obj, x=F('x')+1)

Using update() like this also has the added advantage of only sending the given fields to the database, as opposed to save() which serializes the entire model (possibly causing race conditions even on fields you didn’t modify).

The need for FOR UPDATE

Where things start to break down is when you need to update a row using more complex code. One option is to use stored procedures, but this is effectively impossible to do while keeping mutli-database compatibility. The other option is to do the computation in Python code. Without FOR UPDATE this puts us back into race-condition territory in the same way as Model.save(). What FOR UPDATE does is to set the same dirty flags that UPDATE and DELETE use before returning the rows. This means that no other transaction can alter it.

Unfortunately the Django ORM doesn’t yet expose FOR UPDATE as part of the query system, but with some creativity we can add it in. Much of the credit for this goes to Alexander Artemenko who wrote the initial version of the helper.

from django.db import models, connections
from django.db.models.query import QuerySet

class ForUpdateQuerySet(QuerySet):
    def for_update(self):
        if 'sqlite' in connections[self.db].settings_dict['ENGINE'].lower():
            # Noop on SQLite since it doesn't support FOR UPDATE
            return self
        sql, params = self.query.get_compiler(self.db).as_sql()
        return self.model._default_manager.raw(sql.rstrip() + ' FOR UPDATE', params)

class ForUpdateManager(models.Manager):
    def get_query_set(self):
        return ForUpdateQuerySet(self.model, using=self._db)

Then all you have to do is inherit from ForUpdateManager in your manager and use for_update() at the end of the filter chain:

qs = Foo.objects.filter(id=1).for_update()
obj = qs[0]
update(obj, x=something_complex())

An example

Our main use case at Atari for all of this is updating a user’s billing information, specifically their next billing date for people on monthly cycles. Due to the nature of MMOs, it is possible that someone could redeem a game-time card at the exact moment that a background task is processing them for monthly billing. At heart this code boils down to:

user = User.objects.filter(id=1).for_update()[0]
new_date = user.next_billing_date + relativedelta(months=1, day=user.next_billing_day)
update(user, next_billing_date=new_date)

This is both transactionally safe and concise, which is just about all you can ask for from a SQL database.

Writing DSLs in Javascript

2010-12-01T00:00:00-08:00

Beginnings

Recently at work I have been building a testing console for our web APIs. It began very simply, largely influenced by the excellent Twitter API console. This design rapidly descended into a sea of hacks. The Hurl style works well for RESTful, or nearly RESTful as in Twitter’s case, APIs however it was not as effective for our primarily JSON-RPC based system. My next approach was to make something more terminal-like, based on the WTerm jQuery plugin. This mapped better to a procedural API, but it still felt very limited. Specifically it needed variables.

eval()^{^w}

Given that this was a web site I already had the not insubstantial power of Javascript at my disposal. My first thought was that I could dynamically generate functions to map to our API, and then simply use eval() to process input. This had two major problems; one, I couldn’t isolate the namespace of the terminal from the rest of the page and two, I had no way to deal with the asynchronous nature of the AJAX calls. To fix the first problem I tried several variations of restricting the execution environment of eval(), but I was unable to find an acceptable, cross-browser solution. To deal with the second I tried various attempts at code generation from the input strings, but again I left empty handed. All in all, while Javascript is a very powerful language for scripting pages, it isn’t the right tool for embedded DSLs.

Parser 1.0

Once I had decided down the path of writing a custom DSL, my first thought turned to parsers. WTerm is already a DSL of sorts, but its parser can be summed up as s.split(). This was enough for their examples, but at the very least I need quoted strings. Given my massive programmer brain, I immediately set forth to write such a beast. Below is the final version of it, with luck it can serve as a warning to others:

// Parse arguments
var raw_buffer = value.substring(value.indexOf(command_name)+command_name.length+1).split('');
var args = [];
var buffer = '';
while(raw_buffer.length > 0) {
    // Advance until non-whitespace
    while(raw_buffer.length && raw_buffer[0] == ' ') raw_buffer.shift();
    if(!raw_buffer.length) break;
    // Is this a quoted string?
    if(raw_buffer[0] == '"' || raw_buffer[0] == "'") {
        var quote = raw_buffer.shift(); // Grab the quote
        while(raw_buffer.length && raw_buffer[0] != quote) buffer += raw_buffer.shift();
        if(raw_buffer.length) raw_buffer.shift(); // Discard closing quote
        args.push(buffer);
        buffer = '';
    } else {
        // Number or unquoted text
        while(raw_buffer.length && raw_buffer[0] != ' ' && raw_buffer[0] != '"' && raw_buffer[0] != "'") buffer += raw_buffer.shift();
        // Try it as a number
        var arg = parseFloat(buffer, 10);
        if(!isNaN(arg)) args.push(arg);
        // fallback to a string
        else args.push(buffer);
        buffer = '';
    }
}

Parser 2.0

Some further Google-fu turned up a very detailed article by Douglas Crockford on Top Down Operator Precedence parsers. The example code is conveniently in Javascript, so I set out to adapt it into a working language. The provided parser is pretty much a working implementation of the bits of Javascript that I needed out of the box, so only a few modifications were needed. For starters I changed how name tokens are handled in advance():

o = scope.find(v);
if(!o) {
    o = scope.define(t);
}

This means that any name token will be created in the scope it appears in. I also didn’t want things like flow control or functions so I altered the returned parser to use:

var s = expression(0);

This provided a solid base to build the DSL on, with Javascript literals ( numbers, strings, objects, and arrays), basic math operators, attribute access, and simple variables.

Interpreter

Crockford’s code only provides a parser, the other half is to have something to execute that parse tree. Below is a snippet of my callback-based function:

switch(node.arity) {
case "literal": cont(node.value); return;
case "name":
    if(!(node.value in variables)) {
        err("Variable \""+node.value+"\" not defined");
        return;
    }
    cont(variables[node.value]);
    return;
case "binary":
    switch(node.value) {
    case "=":
        interpret(node.second, function(value) {
            variables[node.first.value] = value;
            cont(value);
        }, err);
        return;
    case "+":
        interpret(node.first, function(value1) {
            interpret(node.second, function(value2) {
               cont(value1 + value2); 
            }, err);
        }, err);
        return;
    }
}

It isn’t the fastest thing on the block, but it is perfectly adequate for interactive usage.

Putting it all together

There are still some bits of complexity to hook up the RPC calls to the interpreter, and in my case setting up OAuth from the browser as well, but once you are this far down the rabbit hole they will be a walk in the park. If you are thinking about building a DSL into a web page, I hope this has pointed you in the right direction, or at least a slightly less wrong one.

Chef slides

2010-11-19T00:00:00-08:00

I will put together a longer post on Chef later, but below are my slides from the SF Django meetup.

System monitoring: A play in 5 acts

2010-11-14T00:00:00-08:00

Act 1: Munin

Munin is dead simple to configure and deploy, and has a very simple, modular sensor API. It uses a collection agent so it can easily monitor values not readable over the network (which is a hard requirement for me). Beyond that it is really just a small wrapper over RRDTool. Its graph UI is painfully simple and it has only the simplest of alerting systems.

Act 2: Munin and Nagios

Nagios is the gold standard when it comes to FOSS network monitoring. It is primarily aimed at alerting on failures, so Munin is still needed for logging time sequence data like performance metrics. This provides more reliable reporting, but it comes with several downsides. One option is to simply duplicate any required sensors between Munin and Nagios. Another is to use the Munin data to trigger alerts in Nagios. This prevents the duplication, but you are still stuck with Munin’s UI for graphing.

Act 3: Nagiosgraph

There are several addons for Nagios that add time-sequence logging, Nagiosgraph and PNP being the most popular. These do require special support from the sensor scripts, but most common plugins will generate the needed output. This does provide a much nicer interface, but it does mean you lose the simplicity of Munin.

Act 4: Zabbix

Zabbix is a young upstart in the monitoring field. It shows a lot of promise as a potential replacement for Nagios, with much simpler configuration and UI. The downside is it doesn’t seem to have any support for logging performance data, so it is a non-starter for now.

Act 5: Opsview

Opsview is built on top of Nagios and Nagiosgraph. It provides some minor UI enhancements as well as some major performance fixes (which I am unlikely to ever notice with ~10 machines to monitor). It also provides somewhat simplified configuration, though the full power of Nagios is available if you dig in to the system.

Conclusion

Opsview seems to be no worse than vanilla Nagios, and it does provide a slightly nicer base to build from. Overall this does feel like picking the lesser of several evils though. The only well-known entry in the Python world is Zenoss, which has minimal remote monitoring capabilities and seems to be even more complex and convoluted than Nagios. Munin 2.0 does show some promise as a future option, but for now Opsview looks like the way to go.