Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

[Link] Neat Python Exceptions; [Link] Fix for Windows 10 to Avoid Ads; [Link] Attacks on ZRTP; [Link] UK Jazz Invasion; [Review] Cuba; [Link] Aricle on Gender Reversal of US Presidential Debate; {OpenSuse] Fix for Network Offline in Updater Applet; [Link] Parkinson's Related to Gut Flora; Farellones Bike Park; [Meta] Tags; Update: Second Ride; Schwalbe Thunder Burt 2.1 v Continental X-King 2.4; Mountain Biking in Santiago; Books on Ethics; Security Fail from Command Driven Interface; Everything Old is New Again; Interesting Take on Trump's Lies; Chutney v6; References on Entropy; Amusing "Alexa.." broadcast; The Shame of Chile's Education System; Playing mp4 gifs in Firefox on Opensuses Leap 42.2; Concurrency at Microsoft; Globalisation: Uk -> Chile; OpenSuse 42.2 and Synaptics Touch-Pads; Even; Cherry Jam; Lebanese Writer Amin Maalouf; C++ - it's the language of the future; Learning From Trump; Chinese Writer Hu Fayun; And; Apricot Jam; Also; Excellent Article on USA Politics; Oh Metafilter; Prejudice Against The Rurals; Also, Zizek; Trump; Why Trump Won; Doxygen + Latex on CentOS 6; SMASH - Solve 5 Biggest Problems in Physics; Good article on racism, brexit, and social divides; Grandaddy are back!; Consciousness From Max Entropy; Democrats; Harvard Will Fix Black Poverty; Modelling Bicycle Wheels; Amusing Polling Outlier; If Labour keeps telling working class people...; Populism and Choice; Books on Defeat; Enrique Ferrari - Argentine Author; Transcript of German Scientists on Learning of Hiroshima; Calvert Journal; Owen Jones on Twitter; Possible Japanese Authors; Complex American Literature; Chutney v5; Weird Componentized Virus; Interesting Argentinian Author - Antonio Di Benedetto; Useful Thread on MetaPhysics; RAND on fighting online anarchy (2001); NSA Hacked; Very Good LRB Article on Brexit; Nussbaum on Anger; Tasting; Apple + Kiwi Jam; Hit Me; Sudoku - CSP + Chaos; Recycling Electronics In Santiago; Vector Displays in OpenGL; And Anti-Aliased; OpenGL - Render via Intermediate Texture; And Garmin Connect; Using Garmin Forerunner 230 With Linux; (Beating Dead Horse) StackOverflow; Current State of Justice in China; Axiom of Determinacy; Ewww; Fee Chaos Book; Course on Differential Geometry; Okay, but...; Sparse Matrices, Deep Learning; Sounds Bad; Applebaum Rape; Tomato Chutney v4; Have to add...; Culturally Liberal and Nothing More; Weird Finite / Infinite Result; Your diamond is a beaten up mess; Maths Books; Good Bike Route from Providencia / Las Condes to Panul; Iain Pears (Author of Complex Plots); Plum Jam; Excellent; More Recently; For a moment I forgot StackOverflow sucked; A Few Weeks On...; Chilean Book Recommendations; How To Write Shared Libraries

© 2006-2017 Andrew Cooke (site) / post authors (content).

Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Wed, 16 Apr 2008 22:23:08 -0400 (CLT)

Below I'll give an example of metaprogramming in Python.  The
motivation is database access, but the approach is very general.


I was thinking of writing a Python program that displays information
from a database, letting the user edit values.  The data have quite a
lot of structure, so I was going to map certain groups of values to
objects.  I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).  So instead I started looking for a way to intercept changes
made to the objects so that I could automatically modify the database.

Initial Thoughts

What I needed then, was a way to intercept changes to attributes (I
didn't want to force the use of "setters" since that is not natural
Python).  It turns out that you can do this using Descriptors, which
are described at

However, the problem with a simple Descriptor approach was that I
ended up repeating myself: attribute "x" needed both a descriptor and
a method to do the database work.  It seemed to me that things would
be simpler if I could somehow generate the Descriptor (ie the
attribute) automatically, using code that recognised the method.

In other words, if a class has the method "set_x" I wanted that to
trigger the generation of an attribute which, using Descriptors, would
call "set_x" when the attribute changed.

I started by looking at factories, but finally it dawned on me that I
what I really wanted was "metaprogramming" and that led me to which, near
the end, has a very nice description of a almost exactly this.  So the
following is pretty much a copy of that.


Below I'll paste the code.  Hopefully the formmating won't go too far

class _AutoWriteDescriptor(object):

    '''Descriptor for AutoWrite.

       Implements an attribute that allows an initial setting then
       calls the setter.

    def __init__(self, name, ignore_identical=True, prefix='_set_'): = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix

    def __get__(self, obj, objtype=None):
        return obj._auto_write_dict[]

    def __set__(self, obj, new_value):
        if in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix,, None)
                obj._auto_write_dict[] = attr(obj, new_value)
            obj._auto_write_dict[] = new_value

    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")

    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                '''AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__'''
                % obj.__class__)

So this (above) is a simple Descriptor.  The only thing to understand
about Descriptors is that they are attributes.  So the methods above
live on the attribute, not on the main object.  When Python accesses
an attribute it checks and, if it's a Descriptor, it invokes the
appropriate method (get if the attribute is being read, set if its
value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way (which is what Python's "new
classes" is all about really) is twofold.  First, you end up uniting
methods and functions (this is emphasised a lot in the Python docs,
but isn't really that interesting, as far as I can see).  Second, you
have an efficient way to change how certain attributes behave.  This
is much better than changing the main object's __getattr__ and
_setattr__ (or whatever thay are called), because the extra code is
only invoked for the "special" attributes - there no speed penalty

I've not said what this Descriptor actually does, but it should be
clear from the discussion earlier.  If the value is being read, the
actual value is pulled from a dictionary that is stored on the main

If the value is being set for the first time (ie when the object is
populated from the database) the value is stored.  Subsequent updates
(ie when the object is modified via the user interface) call the
setter method.  The setter method is "_set_..." (the prefix variable).

class _AutoWriteMeta(type):

    '''Metaclass for AutoWrite.

       Constructs the attributes for AutoWrite using _AutoWriteDescriptor.

    def __init__(cls, name, bases, dict):
        super(_AutoWriteMeta, cls).__init__(name, bases, dict)
        prefix = dict.get('_auto_write_prefix', '_set_')
        ignore_identical = dict.get('_auto_write_ignore_identical', True)
        for attr in dict.keys():
            if attr.startswith(prefix):
                name = attr[len(prefix):]
                setattr(cls, name,
                        _AutoWriteDescriptor(name, ignore_identical, prefix))

This (above) is the metaclass - effectively is a class factory and
it's called when the class description is parsed by Python.  Once you
know that (and assuming you know that everything in Python is
basically a dictionary) you can predict the rest: the dictionary of
methods is read and for each one that starts with "_set" a Descriptor
is generated.

class AutoWrite(object):

    '''Superclass for active attributes.

       Subclasses should define _set_'name' for active attributes.  The
       named attributes are then automatically generated with the
       following behaviour:
       - Initial setting stores the value (typically the initial value is
         stored during creation, reading from a database)
       - Setting a new value that is not equal to the old value will call
         the set method.  That method should either return a value (which is
         what will be stored) or throw GlobalStateChanged, to indicate that
         the entire system needs to re-read values.
       The _set_'name' method should have the signature (self, new_value).

       The ignore_identical and prefix options can be set via the class
       dictionary.  For example, to use method names like 'on_change_x'
       which are called always (even if the value being set is identical):

           class MyClass(AutoWrite):
               _auto_write_prefix = 'on_change_'
               _auto_write_ignore_identical = False

    __metaclass__ = _AutoWriteMeta

    def __init__(self):
        self._auto_write_dict = {}

The Descriptor and metaclass are all that you need, really.  This base
class is just a nice way of putting everything together.  It ensures
that the right metaclass is used and that the dictionary used to store
the values is created.  So to use everything above you just subclass


The following tests show this in action:

class Simple(AutoWrite):

    def __init__(self):
        self.last_setter_called = None

    def _set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

    def _set_inc(self, new_value):
        self.last_setter_called = new_value
        return new_value + 1

class BadInheritance(AutoWrite):

    def __init__(self):

    def _set_x(self, new_value):

class Options(AutoWrite):

    _auto_write_prefix = 'set_'
    _auto_write_ignore_identical = False

    def __init__(self):
        self.last_setter_called = None

    def set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

class AutoWriteTest(unittest.TestCase):

    def test_direct(self):
        simple = Simple()
        simple.x = 1
        self.assertEqual(simple.x, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.x = 2
        self.assertEqual(simple.x, 2)
        self.assertEqual(simple.last_setter_called, 2)
            del simple.x
  "no exception when deleting attribute")
        except AttributeError:
        self.assertEqual(simple.x, 2)

    def test_inc(self):
        simple = Simple() = 1
        self.assertEqual(, 1)
        self.assertEqual(simple.last_setter_called, None) = 2
        self.assertEqual(, 3)
        self.assertEqual(simple.last_setter_called, 2)
  "no exception when deleting attribute")
        except AttributeError:
        self.assertEqual(, 3)

    def test_bad_inheritance(self):
        bad = BadInheritance()
            bad.x = 1
  "expected failure due to no __init__ call")
        except AttributeError, e:
                e.message.find("AutoWrite dictionary missing"), -1)
            self.assertNotEqual(e.message.find("BadInheritance"), -1)

    def test_options(self):
        options = Options()
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, None)
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, 1)

The test_direct method above shows what is happening.  Setting "x" to
1 the first time works like any normal attribute.  The value can be
read back as expected.  Setting it again (to 2) triggers the callback
which, in this case, just sets the attribute "last_setter_called".

Hope that made sense,

Useful Responses to Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Thu, 17 Apr 2008 09:00:30 -0400 (CLT)

There are some useful comments here -

I will update my own code, but not the post here (this clunky web site /
mail archive isn't that easy to update....)


Another, Simpler Python Meta-Programming Example

From: "andrew cooke" <andrew@...>

Date: Tue, 5 Aug 2008 21:24:38 -0400 (CLT)

Here I wanted each class to display self._text in a different way:

class _LexicalMeta(type):
    Constructs __str__ using _template

    def __init__(cls, name, bases, dict):
        super(_LexicalMeta, cls).__init__(name, bases, dict)
        template = dict['_template']
        setattr(cls, '__str__', lambda self: template % self._text)

class _BaseLexical(object):
    Base class for lexical objects.

    __metaclass__ = _LexicalMeta
    _template = None

    def __init__(self, text):
        self._text = text

class Word(_BaseLexical):
    _template = '"%s"'

class Float(_BaseLexical):
    _template = 'float:%s'

class Integer(_BaseLexical):
    _template = 'integer:%s'

class Symbol(_BaseLexical):
    _template = '%s'

So, for example,
  str(Work('abc')) == '"abc"'
  str(Integer('123')) == 'integer:123'


Comment on this post