| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route; Mapa de Ciclovias en Santiago; How Unreliable is UDP?; SE Santiago 20m Bike Route; Cameron's Rap; Configuring libxml with Eclipse; Reducing Combinatorial Complexity With Occam - AI; Sentidos Comunes (Chilean Online Magazine); Hilary Mantel: The Assassination of Margaret Thatcher - August 6th 1983; NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls; Formalising NSA Attacks; Switching Brakes (Tektro Hydraulic); Naim NAP 100 (Power Amp); AKG 550 First Impressions; Facebook manipulates emotions (no really); Map Reduce "No Longer Used" At Google; Removing RAID metadata; New Bike (Good Bike Shop, Santiago Chile); Removing APE Tags in Linux; Compiling Python 3.0 With GCC 4.8; Maven is Amazing; Generating Docs from a GitHub Wiki; Modular Shelves; Bash Best Practices; Good Emergency Gasfiter (Santiago, Chile); Readings in Recent Architecture; Roger Casement; Integrated Information Theory (Or Not); Possibly undefined macro AC_ENABLE_SHARED; Update on Charges; Sunburst Visualisation; Spectral Embeddings (Distances -> Coordinates); Introduction to Causality; Filtering To Help Colour-Blindness; ASUS 1015E-DS02 Too; Ready Player One; Writing Clear, Fast Julia Code; List of LatAm Novels; Running (for women); Building a Jenkins Plugin and a Jar (for Command Line use); Headphone Test Recordings; Causal Consistency; The Quest for Randomness; Chat Wars; Real-life Financial Co Without ACID Database...

© 2006-2013 Andrew Cooke (site) / post authors (content).

Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Wed, 16 Apr 2008 22:23:08 -0400 (CLT)

Below I'll give an example of metaprogramming in Python.  The
motivation is database access, but the approach is very general.


I was thinking of writing a Python program that displays information
from a database, letting the user edit values.  The data have quite a
lot of structure, so I was going to map certain groups of values to
objects.  I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).  So instead I started looking for a way to intercept changes
made to the objects so that I could automatically modify the database.

Initial Thoughts

What I needed then, was a way to intercept changes to attributes (I
didn't want to force the use of "setters" since that is not natural
Python).  It turns out that you can do this using Descriptors, which
are described at http://users.rcn.com/python/download/Descriptor.htm

However, the problem with a simple Descriptor approach was that I
ended up repeating myself: attribute "x" needed both a descriptor and
a method to do the database work.  It seemed to me that things would
be simpler if I could somehow generate the Descriptor (ie the
attribute) automatically, using code that recognised the method.

In other words, if a class has the method "set_x" I wanted that to
trigger the generation of an attribute which, using Descriptors, would
call "set_x" when the attribute changed.

I started by looking at factories, but finally it dawned on me that I
what I really wanted was "metaprogramming" and that led me to
http://www.python.org/download/releases/2.2.3/descrintro/ which, near
the end, has a very nice description of a almost exactly this.  So the
following is pretty much a copy of that.


Below I'll paste the code.  Hopefully the formmating won't go too far

class _AutoWriteDescriptor(object):

    '''Descriptor for AutoWrite.

       Implements an attribute that allows an initial setting then
       calls the setter.

    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix

    def __get__(self, obj, objtype=None):
        return obj._auto_write_dict[self.name]

    def __set__(self, obj, new_value):
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
            obj._auto_write_dict[self.name] = new_value

    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")

    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                '''AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__'''
                % obj.__class__)

So this (above) is a simple Descriptor.  The only thing to understand
about Descriptors is that they are attributes.  So the methods above
live on the attribute, not on the main object.  When Python accesses
an attribute it checks and, if it's a Descriptor, it invokes the
appropriate method (get if the attribute is being read, set if its
value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way (which is what Python's "new
classes" is all about really) is twofold.  First, you end up uniting
methods and functions (this is emphasised a lot in the Python docs,
but isn't really that interesting, as far as I can see).  Second, you
have an efficient way to change how certain attributes behave.  This
is much better than changing the main object's __getattr__ and
_setattr__ (or whatever thay are called), because the extra code is
only invoked for the "special" attributes - there no speed penalty

I've not said what this Descriptor actually does, but it should be
clear from the discussion earlier.  If the value is being read, the
actual value is pulled from a dictionary that is stored on the main

If the value is being set for the first time (ie when the object is
populated from the database) the value is stored.  Subsequent updates
(ie when the object is modified via the user interface) call the
setter method.  The setter method is "_set_..." (the prefix variable).

class _AutoWriteMeta(type):

    '''Metaclass for AutoWrite.

       Constructs the attributes for AutoWrite using _AutoWriteDescriptor.

    def __init__(cls, name, bases, dict):
        super(_AutoWriteMeta, cls).__init__(name, bases, dict)
        prefix = dict.get('_auto_write_prefix', '_set_')
        ignore_identical = dict.get('_auto_write_ignore_identical', True)
        for attr in dict.keys():
            if attr.startswith(prefix):
                name = attr[len(prefix):]
                setattr(cls, name,
                        _AutoWriteDescriptor(name, ignore_identical, prefix))

This (above) is the metaclass - effectively is a class factory and
it's called when the class description is parsed by Python.  Once you
know that (and assuming you know that everything in Python is
basically a dictionary) you can predict the rest: the dictionary of
methods is read and for each one that starts with "_set" a Descriptor
is generated.

class AutoWrite(object):

    '''Superclass for active attributes.

       Subclasses should define _set_'name' for active attributes.  The
       named attributes are then automatically generated with the
       following behaviour:
       - Initial setting stores the value (typically the initial value is
         stored during creation, reading from a database)
       - Setting a new value that is not equal to the old value will call
         the set method.  That method should either return a value (which is
         what will be stored) or throw GlobalStateChanged, to indicate that
         the entire system needs to re-read values.
       The _set_'name' method should have the signature (self, new_value).

       The ignore_identical and prefix options can be set via the class
       dictionary.  For example, to use method names like 'on_change_x'
       which are called always (even if the value being set is identical):

           class MyClass(AutoWrite):
               _auto_write_prefix = 'on_change_'
               _auto_write_ignore_identical = False

    __metaclass__ = _AutoWriteMeta

    def __init__(self):
        self._auto_write_dict = {}

The Descriptor and metaclass are all that you need, really.  This base
class is just a nice way of putting everything together.  It ensures
that the right metaclass is used and that the dictionary used to store
the values is created.  So to use everything above you just subclass


The following tests show this in action:

class Simple(AutoWrite):

    def __init__(self):
        self.last_setter_called = None

    def _set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

    def _set_inc(self, new_value):
        self.last_setter_called = new_value
        return new_value + 1

class BadInheritance(AutoWrite):

    def __init__(self):

    def _set_x(self, new_value):

class Options(AutoWrite):

    _auto_write_prefix = 'set_'
    _auto_write_ignore_identical = False

    def __init__(self):
        self.last_setter_called = None

    def set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

class AutoWriteTest(unittest.TestCase):

    def test_direct(self):
        simple = Simple()
        simple.x = 1
        self.assertEqual(simple.x, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.x = 2
        self.assertEqual(simple.x, 2)
        self.assertEqual(simple.last_setter_called, 2)
            del simple.x
            self.fail("no exception when deleting attribute")
        except AttributeError:
        self.assertEqual(simple.x, 2)

    def test_inc(self):
        simple = Simple()
        simple.inc = 1
        self.assertEqual(simple.inc, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.inc = 2
        self.assertEqual(simple.inc, 3)
        self.assertEqual(simple.last_setter_called, 2)
            del simple.inc
            self.fail("no exception when deleting attribute")
        except AttributeError:
        self.assertEqual(simple.inc, 3)

    def test_bad_inheritance(self):
        bad = BadInheritance()
            bad.x = 1
            self.fail("expected failure due to no __init__ call")
        except AttributeError, e:
                e.message.find("AutoWrite dictionary missing"), -1)
            self.assertNotEqual(e.message.find("BadInheritance"), -1)

    def test_options(self):
        options = Options()
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, None)
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, 1)

The test_direct method above shows what is happening.  Setting "x" to
1 the first time works like any normal attribute.  The value can be
read back as expected.  Setting it again (to 2) triggers the callback
which, in this case, just sets the attribute "last_setter_called".

Hope that made sense,

Useful Responses to Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Thu, 17 Apr 2008 09:00:30 -0400 (CLT)

There are some useful comments here -

I will update my own code, but not the post here (this clunky web site /
mail archive isn't that easy to update....)


Another, Simpler Python Meta-Programming Example

From: "andrew cooke" <andrew@...>

Date: Tue, 5 Aug 2008 21:24:38 -0400 (CLT)

Here I wanted each class to display self._text in a different way:

class _LexicalMeta(type):
    Constructs __str__ using _template

    def __init__(cls, name, bases, dict):
        super(_LexicalMeta, cls).__init__(name, bases, dict)
        template = dict['_template']
        setattr(cls, '__str__', lambda self: template % self._text)

class _BaseLexical(object):
    Base class for lexical objects.

    __metaclass__ = _LexicalMeta
    _template = None

    def __init__(self, text):
        self._text = text

class Word(_BaseLexical):
    _template = '"%s"'

class Float(_BaseLexical):
    _template = 'float:%s'

class Integer(_BaseLexical):
    _template = 'integer:%s'

class Symbol(_BaseLexical):
    _template = '%s'

So, for example,
  str(Work('abc')) == '"abc"'
  str(Integer('123')) == 'integer:123'


Comment on this post