Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Wed, 16 Apr 2008 22:23:08 -0400 (CLT)

Below I'll give an example of metaprogramming in Python.  The
motivation is database access, but the approach is very general.


Motivation
----------

I was thinking of writing a Python program that displays information
from a database, letting the user edit values.  The data have quite a
lot of structure, so I was going to map certain groups of values to
objects.  I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).  So instead I started looking for a way to intercept changes
made to the objects so that I could automatically modify the database.


Initial Thoughts
----------------

What I needed then, was a way to intercept changes to attributes (I
didn't want to force the use of "setters" since that is not natural
Python).  It turns out that you can do this using Descriptors, which
are described at http://users.rcn.com/python/download/Descriptor.htm

However, the problem with a simple Descriptor approach was that I
ended up repeating myself: attribute "x" needed both a descriptor and
a method to do the database work.  It seemed to me that things would
be simpler if I could somehow generate the Descriptor (ie the
attribute) automatically, using code that recognised the method.

In other words, if a class has the method "set_x" I wanted that to
trigger the generation of an attribute which, using Descriptors, would
call "set_x" when the attribute changed.

I started by looking at factories, but finally it dawned on me that I
what I really wanted was "metaprogramming" and that led me to
http://www.python.org/download/releases/2.2.3/descrintro/ which, near
the end, has a very nice description of a almost exactly this.  So the
following is pretty much a copy of that.


Implementation
--------------

Below I'll paste the code.  Hopefully the formmating won't go too far
wrong...


class _AutoWriteDescriptor(object):

    '''Descriptor for AutoWrite.

       Implements an attribute that allows an initial setting then
       calls the setter.
    '''

    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix

    def __get__(self, obj, objtype=None):
        self.check_dictionary(obj)
        return obj._auto_write_dict[self.name]

    def __set__(self, obj, new_value):
        self.check_dictionary(obj)
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
        else:
            obj._auto_write_dict[self.name] = new_value

    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")

    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                '''AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__'''
                % obj.__class__)

So this (above) is a simple Descriptor.  The only thing to understand
about Descriptors is that they are attributes.  So the methods above
live on the attribute, not on the main object.  When Python accesses
an attribute it checks and, if it's a Descriptor, it invokes the
appropriate method (get if the attribute is being read, set if its
value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way (which is what Python's "new
classes" is all about really) is twofold.  First, you end up uniting
methods and functions (this is emphasised a lot in the Python docs,
but isn't really that interesting, as far as I can see).  Second, you
have an efficient way to change how certain attributes behave.  This
is much better than changing the main object's __getattr__ and
_setattr__ (or whatever thay are called), because the extra code is
only invoked for the "special" attributes - there no speed penalty
otherwise.

I've not said what this Descriptor actually does, but it should be
clear from the discussion earlier.  If the value is being read, the
actual value is pulled from a dictionary that is stored on the main
object.

If the value is being set for the first time (ie when the object is
populated from the database) the value is stored.  Subsequent updates
(ie when the object is modified via the user interface) call the
setter method.  The setter method is "_set_..." (the prefix variable).


class _AutoWriteMeta(type):

    '''Metaclass for AutoWrite.

       Constructs the attributes for AutoWrite using _AutoWriteDescriptor.
    '''

    def __init__(cls, name, bases, dict):
        super(_AutoWriteMeta, cls).__init__(name, bases, dict)
        prefix = dict.get('_auto_write_prefix', '_set_')
        ignore_identical = dict.get('_auto_write_ignore_identical', True)
        for attr in dict.keys():
            if attr.startswith(prefix):
                name = attr[len(prefix):]
                setattr(cls, name,
                        _AutoWriteDescriptor(name, ignore_identical, prefix))

This (above) is the metaclass - effectively is a class factory and
it's called when the class description is parsed by Python.  Once you
know that (and assuming you know that everything in Python is
basically a dictionary) you can predict the rest: the dictionary of
methods is read and for each one that starts with "_set" a Descriptor
is generated.


class AutoWrite(object):

    '''Superclass for active attributes.

       Subclasses should define _set_'name' for active attributes.  The
       named attributes are then automatically generated with the
       following behaviour:
       - Initial setting stores the value (typically the initial value is
         stored during creation, reading from a database)
       - Setting a new value that is not equal to the old value will call
         the set method.  That method should either return a value (which is
         what will be stored) or throw GlobalStateChanged, to indicate that
         the entire system needs to re-read values.
       The _set_'name' method should have the signature (self, new_value).

       The ignore_identical and prefix options can be set via the class
       dictionary.  For example, to use method names like 'on_change_x'
       which are called always (even if the value being set is identical):

           class MyClass(AutoWrite):
               _auto_write_prefix = 'on_change_'
               _auto_write_ignore_identical = False
    '''

    __metaclass__ = _AutoWriteMeta

    def __init__(self):
        self._auto_write_dict = {}

The Descriptor and metaclass are all that you need, really.  This base
class is just a nice way of putting everything together.  It ensures
that the right metaclass is used and that the dictionary used to store
the values is created.  So to use everything above you just subclass
this.


Tests
-----

The following tests show this in action:

class Simple(AutoWrite):

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def _set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

    def _set_inc(self, new_value):
        self.last_setter_called = new_value
        return new_value + 1


class BadInheritance(AutoWrite):

    def __init__(self):
        pass

    def _set_x(self, new_value):
        pass


class Options(AutoWrite):

    _auto_write_prefix = 'set_'
    _auto_write_ignore_identical = False

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value


class AutoWriteTest(unittest.TestCase):

    def test_direct(self):
        simple = Simple()
        simple.x = 1
        self.assertEqual(simple.x, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.x = 2
        self.assertEqual(simple.x, 2)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.x
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.x, 2)

    def test_inc(self):
        simple = Simple()
        simple.inc = 1
        self.assertEqual(simple.inc, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.inc = 2
        self.assertEqual(simple.inc, 3)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.inc
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.inc, 3)

    def test_bad_inheritance(self):
        bad = BadInheritance()
        try:
            bad.x = 1
            self.fail("expected failure due to no __init__ call")
        except AttributeError, e:
            self.assertNotEqual(
                e.message.find("AutoWrite dictionary missing"), -1)
            self.assertNotEqual(e.message.find("BadInheritance"), -1)

    def test_options(self):
        options = Options()
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, None)
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, 1)

The test_direct method above shows what is happening.  Setting "x" to
1 the first time works like any normal attribute.  The value can be
read back as expected.  Setting it again (to 2) triggers the callback
which, in this case, just sets the attribute "last_setter_called".


Hope that made sense,
Andrew

There are some useful comments here -
http://groups.google.com/group/comp.lang.python/browse_thread/thread/e4144d9c8fafe29a

I will update my own code, but not the post here (this clunky web site /
mail archive isn't that easy to update....)

Andrew