| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls; Formalising NSA Attacks; Switching Brakes (Tektro Hydraulic); Naim NAP 100 (Power Amp); AKG 550 First Impressions; Facebook manipulates emotions (no really); Map Reduce "No Longer Used" At Google; Removing RAID metadata; New Bike (Good Bike Shop, Santiago Chile); Removing APE Tags in Linux; Compiling Python 3.0 With GCC 4.8; Maven is Amazing; Generating Docs from a GitHub Wiki; Modular Shelves; Bash Best Practices; Good Emergency Gasfiter (Santiago, Chile); Readings in Recent Architecture; Roger Casement; Integrated Information Theory (Or Not); Possibly undefined macro AC_ENABLE_SHARED; Update on Charges; Sunburst Visualisation; Spectral Embeddings (Distances -> Coordinates); Introduction to Causality; Filtering To Help Colour-Blindness; ASUS 1015E-DS02 Too; Ready Player One; Writing Clear, Fast Julia Code; List of LatAm Novels; Running (for women); Building a Jenkins Plugin and a Jar (for Command Line use); Headphone Test Recordings; Causal Consistency; The Quest for Randomness; Chat Wars; Real-life Financial Co Without ACID Database...; Flexible Muscle-Based Locomotion for Bipedal Creatures; SQL Performance Explained; The Little Manual of API Design; Multiple Word Sizes; CRC - Next Steps; FizzBuzz; Update on CRCs; Decent Links / Discussion Community; Automated Reasoning About LLVM Optimizations and Undefined Behavior; A Painless Guide To CRC Error Detection Algorithms; Tests in Julia; Dave Eggers: what's so funny about peace, love and Starship?; Cello - High Level C Programming; autoreconf needs tar; Will Self Goes To Heathrow; Top 5 BioInformatics Papers; Vasovagal Response; Good Food in Vina; Chilean Drug Criminals Use Subsitution Cipher; Adrenaline; Stiglitz on the Impact of Technology; Why Not; How I Am 5; Lenovo X240 OpenSuse 13.1; NSA and GCHQ - Psychological Trolls; Finite Fields in Julia (Defining Your Own Number Type); Julian Assange; Starting Qemu on OpenSuse; Noisy GAs/TMs; Venezuela; Reinstalling GRUB with EFI; Instructions For Disabling KDE Indexing; Evolving Speakers; Changing Salt Size in Simple Crypt 3.0.0; Logarithmic Map (Moved); More Info; Words Found in Voynich Manuscript; An Inventory Of 3D Space-Filling Curves; Foxes Using Magnetic Fields To Hunt; 5 Rounds RC5 No Rotation; JP Morgan and Madoff; Ori - Secure, Distributed File System; Physical Unclonable Functions (PUFs); Prejudice on Reddit; Recursion OK; Optimizing Julia Code

© 2006-2013 Andrew Cooke (site) / post authors (content).

Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Wed, 16 Apr 2008 22:23:08 -0400 (CLT)

Below I'll give an example of metaprogramming in Python.  The
motivation is database access, but the approach is very general.


Motivation
----------

I was thinking of writing a Python program that displays information
from a database, letting the user edit values.  The data have quite a
lot of structure, so I was going to map certain groups of values to
objects.  I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).  So instead I started looking for a way to intercept changes
made to the objects so that I could automatically modify the database.


Initial Thoughts
----------------

What I needed then, was a way to intercept changes to attributes (I
didn't want to force the use of "setters" since that is not natural
Python).  It turns out that you can do this using Descriptors, which
are described at http://users.rcn.com/python/download/Descriptor.htm

However, the problem with a simple Descriptor approach was that I
ended up repeating myself: attribute "x" needed both a descriptor and
a method to do the database work.  It seemed to me that things would
be simpler if I could somehow generate the Descriptor (ie the
attribute) automatically, using code that recognised the method.

In other words, if a class has the method "set_x" I wanted that to
trigger the generation of an attribute which, using Descriptors, would
call "set_x" when the attribute changed.

I started by looking at factories, but finally it dawned on me that I
what I really wanted was "metaprogramming" and that led me to
http://www.python.org/download/releases/2.2.3/descrintro/ which, near
the end, has a very nice description of a almost exactly this.  So the
following is pretty much a copy of that.


Implementation
--------------

Below I'll paste the code.  Hopefully the formmating won't go too far
wrong...


class _AutoWriteDescriptor(object):

    '''Descriptor for AutoWrite.

       Implements an attribute that allows an initial setting then
       calls the setter.
    '''

    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix

    def __get__(self, obj, objtype=None):
        self.check_dictionary(obj)
        return obj._auto_write_dict[self.name]

    def __set__(self, obj, new_value):
        self.check_dictionary(obj)
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
        else:
            obj._auto_write_dict[self.name] = new_value

    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")

    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                '''AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__'''
                % obj.__class__)

So this (above) is a simple Descriptor.  The only thing to understand
about Descriptors is that they are attributes.  So the methods above
live on the attribute, not on the main object.  When Python accesses
an attribute it checks and, if it's a Descriptor, it invokes the
appropriate method (get if the attribute is being read, set if its
value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way (which is what Python's "new
classes" is all about really) is twofold.  First, you end up uniting
methods and functions (this is emphasised a lot in the Python docs,
but isn't really that interesting, as far as I can see).  Second, you
have an efficient way to change how certain attributes behave.  This
is much better than changing the main object's __getattr__ and
_setattr__ (or whatever thay are called), because the extra code is
only invoked for the "special" attributes - there no speed penalty
otherwise.

I've not said what this Descriptor actually does, but it should be
clear from the discussion earlier.  If the value is being read, the
actual value is pulled from a dictionary that is stored on the main
object.

If the value is being set for the first time (ie when the object is
populated from the database) the value is stored.  Subsequent updates
(ie when the object is modified via the user interface) call the
setter method.  The setter method is "_set_..." (the prefix variable).


class _AutoWriteMeta(type):

    '''Metaclass for AutoWrite.

       Constructs the attributes for AutoWrite using _AutoWriteDescriptor.
    '''

    def __init__(cls, name, bases, dict):
        super(_AutoWriteMeta, cls).__init__(name, bases, dict)
        prefix = dict.get('_auto_write_prefix', '_set_')
        ignore_identical = dict.get('_auto_write_ignore_identical', True)
        for attr in dict.keys():
            if attr.startswith(prefix):
                name = attr[len(prefix):]
                setattr(cls, name,
                        _AutoWriteDescriptor(name, ignore_identical, prefix))

This (above) is the metaclass - effectively is a class factory and
it's called when the class description is parsed by Python.  Once you
know that (and assuming you know that everything in Python is
basically a dictionary) you can predict the rest: the dictionary of
methods is read and for each one that starts with "_set" a Descriptor
is generated.


class AutoWrite(object):

    '''Superclass for active attributes.

       Subclasses should define _set_'name' for active attributes.  The
       named attributes are then automatically generated with the
       following behaviour:
       - Initial setting stores the value (typically the initial value is
         stored during creation, reading from a database)
       - Setting a new value that is not equal to the old value will call
         the set method.  That method should either return a value (which is
         what will be stored) or throw GlobalStateChanged, to indicate that
         the entire system needs to re-read values.
       The _set_'name' method should have the signature (self, new_value).

       The ignore_identical and prefix options can be set via the class
       dictionary.  For example, to use method names like 'on_change_x'
       which are called always (even if the value being set is identical):

           class MyClass(AutoWrite):
               _auto_write_prefix = 'on_change_'
               _auto_write_ignore_identical = False
    '''

    __metaclass__ = _AutoWriteMeta

    def __init__(self):
        self._auto_write_dict = {}

The Descriptor and metaclass are all that you need, really.  This base
class is just a nice way of putting everything together.  It ensures
that the right metaclass is used and that the dictionary used to store
the values is created.  So to use everything above you just subclass
this.


Tests
-----

The following tests show this in action:

class Simple(AutoWrite):

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def _set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

    def _set_inc(self, new_value):
        self.last_setter_called = new_value
        return new_value + 1


class BadInheritance(AutoWrite):

    def __init__(self):
        pass

    def _set_x(self, new_value):
        pass


class Options(AutoWrite):

    _auto_write_prefix = 'set_'
    _auto_write_ignore_identical = False

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value


class AutoWriteTest(unittest.TestCase):

    def test_direct(self):
        simple = Simple()
        simple.x = 1
        self.assertEqual(simple.x, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.x = 2
        self.assertEqual(simple.x, 2)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.x
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.x, 2)

    def test_inc(self):
        simple = Simple()
        simple.inc = 1
        self.assertEqual(simple.inc, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.inc = 2
        self.assertEqual(simple.inc, 3)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.inc
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.inc, 3)

    def test_bad_inheritance(self):
        bad = BadInheritance()
        try:
            bad.x = 1
            self.fail("expected failure due to no __init__ call")
        except AttributeError, e:
            self.assertNotEqual(
                e.message.find("AutoWrite dictionary missing"), -1)
            self.assertNotEqual(e.message.find("BadInheritance"), -1)

    def test_options(self):
        options = Options()
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, None)
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, 1)

The test_direct method above shows what is happening.  Setting "x" to
1 the first time works like any normal attribute.  The value can be
read back as expected.  Setting it again (to 2) triggers the callback
which, in this case, just sets the attribute "last_setter_called".


Hope that made sense,
Andrew

Useful Responses to Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Thu, 17 Apr 2008 09:00:30 -0400 (CLT)

There are some useful comments here -
http://groups.google.com/group/comp.lang.python/browse_thread/thread/e4144d9c8fafe29a

I will update my own code, but not the post here (this clunky web site /
mail archive isn't that easy to update....)

Andrew

Another, Simpler Python Meta-Programming Example

From: "andrew cooke" <andrew@...>

Date: Tue, 5 Aug 2008 21:24:38 -0400 (CLT)

Here I wanted each class to display self._text in a different way:

class _LexicalMeta(type):
    '''
    Constructs __str__ using _template
    '''

    def __init__(cls, name, bases, dict):
        super(_LexicalMeta, cls).__init__(name, bases, dict)
        template = dict['_template']
        setattr(cls, '__str__', lambda self: template % self._text)


class _BaseLexical(object):
    '''
    Base class for lexical objects.
    '''

    __metaclass__ = _LexicalMeta
    _template = None

    def __init__(self, text):
        self._text = text


class Word(_BaseLexical):
    _template = '"%s"'

class Float(_BaseLexical):
    _template = 'float:%s'

class Integer(_BaseLexical):
    _template = 'integer:%s'

class Symbol(_BaseLexical):
    _template = '%s'


So, for example,
  str(Work('abc')) == '"abc"'
  str(Integer('123')) == 'integer:123'

Andrew

Comment on this post