| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

Chinese Govt Intercepts External Web To DDOS github; Numbering Permutations; Teenage Engineering - Low Price Synths; GCHQ Can Do Whatever It Wants; Dublinesque; A Cryptographic SAT Solver; Security Challenges; Word Lists for Crosswords; 3D Printing and Speaker Design; Searchable Snowden Archive; XCode Backdoored; Derived Apps Have Malware (CIA); Rowhammer - Hacking Software Via Hardware (DRAM) Bugs; Immutable SQL Database (Kinda); Tor GPS Tracker; That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route; Mapa de Ciclovias en Santiago; How Unreliable is UDP?; SE Santiago 20m Bike Route; Cameron's Rap; Configuring libxml with Eclipse; Reducing Combinatorial Complexity With Occam - AI; Sentidos Comunes (Chilean Online Magazine); Hilary Mantel: The Assassination of Margaret Thatcher - August 6th 1983; NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls; Formalising NSA Attacks; Switching Brakes (Tektro Hydraulic); Naim NAP 100 (Power Amp); AKG 550 First Impressions; Facebook manipulates emotions (no really); Map Reduce "No Longer Used" At Google; Removing RAID metadata; New Bike (Good Bike Shop, Santiago Chile); Removing APE Tags in Linux; Compiling Python 3.0 With GCC 4.8; Maven is Amazing; Generating Docs from a GitHub Wiki; Modular Shelves; Bash Best Practices; Good Emergency Gasfiter (Santiago, Chile); Readings in Recent Architecture; Roger Casement; Integrated Information Theory (Or Not); Possibly undefined macro AC_ENABLE_SHARED; Update on Charges; Sunburst Visualisation

© 2006-2013 Andrew Cooke (site) / post authors (content).

iBatis ORM and Caching Strategy - a Use Case

From: "andrew cooke" <andrew@...>

Date: Sat, 27 Sep 2008 04:34:37 -0400 (CLT)

The Application

I am writing a Java application that presents information, stored in a
database, via a web server.  The presentation does not modify the data,
but the database does evolve, slowly, on disk (modified by other

The data themselves can be divided into two groups: first a set of related
objects that describe the system being measured (in this context they
could be considered "metadata"); second a set of measurements.  There are
many more measurements than there are metadata.

Some iBatis Details

iBatis provides very simple caching at the SQL statement level.  So if a
query is repeated with the same parameters then the result may be
retrieved from the cache.  The advantage of this approach is that it is
simple to implement and understand.  The disadvantage is that it is not
"intelligent", in that it does not recognise object identities.

For example, if I have two different queries that both return an instance
of the same class, iBatis will cache two different instances of the same
object for each key - one from each query.

iBatis is also limited in how it will map graphs - it does not provide
transparent retrieval of related objects.  So if one class references
another class then either both must be mapped in a single query to iBatis,
or a second call must be made explicitly (in which case the first instance
presumably contains a key that will be used to find the correct value).

This limitation can be removed by using a second layer - for example
Spring AOP.

My Solution

I have decided to use a very simple mapping approach with two levels of
cache.  The first level of cache is the iBatis statement cache; the second
is a set of maps from keys to instances for the "metadata" objects.  This
second level holds all metadata instances in-memory.  The caches are in a
singleton "database interface" class.

The connections between related objects are not explicit in the objects
themselves.  Instead, the "database interface" provides methods that
retrieve related objects (for example "List<Foo> getFoosForBaz(Baz)").

One drawback to this approach is that it is not very "OO".  The instances
do not form a graph that can be traversed directly.  For this particular
application that is not an issue - there is very little "analysis" of the
data (it is just "dumb presentation").

Another drawback is that for each metadata class the ORM manages two
different types - instances and keys.  iBatis must return instances when
asked for "all instances" to populate the cache, but must return only keys
when asked about relations between objects (the keys are then used to
retrieve instances from the second level cache).  In practice, with some
basic engineering (return factory objects rather than keys; these take the
"database interface" and extract the cached value), the extra work is

The advantages are that the scheme is fast, compact, and relatively easy
to understand.  It is fast because almost all information is cached:
metadata instances are cached in the second level caches; relationships
are cached (as lists of keys) in the iBatis cache.  It is compact because
no duplicate instances are created (except when a cache expires).

The simplicity is not so obvious until you start to consider details like
cache expiry.  Because there are no explicit relations each pool of
instances (caches are grouped by class) can expire independently.  There
is no worry about instances being "trapped" in complex graphs.  There
should be no memory leaks.

This approach is very different to how I used SQLAlchemy in Python.  The
approach I used there was much more "sophisticated", with transparent,
lazy retrieval of related objects.  In that case I was writing a client
application that did not need to be long-lived, reliable, or efficient
(within reason, of course).

It's possible that I am being too conservative in this case.  Spring AOP
(or Hibernate) might have made the code more transparent at no extra cost
in practice.  But I think this was a reasonable approach to use here,
given the circumstances (both the application - particularly the "read
only" nature - and my limited knowledge).


More iBatis Comments

From: "andrew cooke" <andrew@...>

Date: Sat, 27 Sep 2008 09:01:17 -0400 (CLT)

When I was first deciding which ORM solution to use for this project I
read, somewhere, a comment suggesting that Hibernate and iBatis were both
good products, and that the choice of which to use depended on how you
wanted to think about the database: Hibernate was best for those who saw
the database as a transparent store for Java code; iBatis was better for
those who wanted to explicitly manage an interface between Java and SQL.

I don't know if that is true, but one more advantage of the approach
described here is that there's no real restriction to follow a "data
model".  Last night I was working through some uses cases for the user
interface and realised that, for "usability", I needed to present a
relationship between the metadata objects that was completely unintuitive
- it would not have been present in a traditional graph of objects and
will require some custom SQL to generate.  That's no problem here.

(You could argue that I simply has a bad design.  I would say that the
intuitive / straw-man data model above is actually a good design and that
this relationship is really an alternative view of the system that is
particularly useful in one context,  So in a more "OO" approach it would
be better considered the result of some analysis by the system (and still
not directly expressed in the data model).

The difference then comes down to how to manage that analysis in an
efficient manner; with the approach outlined here I can place the analysis
in the database and rely on the existing caches.  If I had been working
with transparently mapped objects I would have been tempted to do the
analysis in Java and add an explicit cache.  From my point of view (ie
preferring to use SQL to do a complex query rather than write Java code)
the approach here is simpler.)

Another possible advantage is the clear separation of concerns.  Quite
naturally my implementation has a single, well-defined interface that
describes all the information I retrieve from the database.  A requirement
that I didn't mention earlier is that the system be easy to move to a
in-house (client) data storage system that is not based on SQL.  Again: no
problem here.

Finally, given the somewhat "crude" (or "hands on") approach I have
described, you may ask whether iBatis is "worth it".  It most certainly
is.  First, it does most of the work of constructing objects for me
(although there is often a final pass to explicitly retrieve objects from
the second level cache).  Second, it allows easy separation of SQL and
Java (in separate files).  Third, it provides a mechanism for adapting the
SQL to different databases (I pass the engine name as a parameter in
queries and that can be used by iBatis to construct the correct query). 
Fourth, it provides the useful first level cache.


Simplified Caching; Problem with iBatis, Spring and OSCache

From: "andrew cooke" <andrew@...>

Date: Sat, 27 Sep 2008 20:36:36 -0400 (CLT)

After writing the above and sleeping on it, I realised that I had made
things unnecessarily complicated.  A single cache (provided by iBatis) is
sufficient.  The only modifications needed are:

- Instead of caching all instances in a second level cache, we
  retrieve instances explicitly via their keys, one by one.  This
  places each instance in the iBatis cache.

- Any query that references an instance returns the key, wrapped in
  a factory as before.  The factory is then passed the database
  interface and requests the instance from the cache.

- The factories are themselves cached by iBatis.  So they can be
  made mutable, storing a weak reference to the instance that was
  retrieved.  If they are then re-used they do not need to repeat
  the lookup (the factory "short circuits" via the stored value).

That required a couple of hours to implement and works fine.

I then decided to investigate how teh caching was working.  To do this I
switched to OSCache and monitored the cache administrator via JMX - see
http://www.opensymphony.com/oscache/wiki/JMX Monitoring.html

That would have worked perfectly, except that iBaits will not use the
OSCache created in Spring.  It always creates its own instance.  After
looking in detail at the iBatis docs (the dev API at
http://ibatis.apache.org/docs/java/dev/index.html ) I realised that this
is a consequence of the caching architecture - the CacheModel creates a
CacheController instance via the constructor.  There is no way to inject a

This is a serious design flaw for iBatis.  It means that caching cannot be
integrated with Spring (there's not even any way to programmatically
retrieve the cache that I can see).


More Info on IBatis-Based Project

From: "andrew cooke" <andrew@...>

Date: Thu, 16 Oct 2008 09:14:37 -0300 (CLST)

I wrote a paper summarising the architecture of the project, including
iBatis.  It's here - http://www.acooke.org/kpi.pdf - and might be useful
as background for the comments above (or as more detail about how I used
iBatis within a J2EE web application).


iBATIS Caching

From: "andrew cooke" <andrew@...>

Date: Sun, 16 Nov 2008 16:00:15 -0300 (CLST)

I was going to add a post summarizing this, but it's simpler to just
include the whole email.  Thanks, Clinton.

---------------------------- Original Message ----------------------------
Subject: iBATIS Caching
From:    "Clinton Begin" <clinton.begin@...>
Date:    Tue, November 4, 2008 12:34 am
To:      andrew@...

Hi Andrew,

A friend of mine forwarded a post on to me and I thought I might offer some

If you'd like iBATIS to use the Spring instance of OSCache, you can create
your own implementation of CacheController.  The interface is quite simple
and self explanatory (but is documented in the javadocs a little more, as
well as the user guide, wiki and our book).

Your cache controller implementation can request the instance of OSCache
from Spring's app context.  It should be quite simple.  To use your
implementation, you can configure it like any other cache.

  <cacheModel id="someCache" type="org.acooke.some.app.cache.CustomOSCache"
readOnly="true" serialize="false">
    <flushInterval hours="24"/>
    <flushOnExecute statement="updateAccountViaInlineParameters"/>
    <property name="size" value="1"/>

Any <property> elements will be passed to setProperties after
instantiation.  You can use a typeAlias to shorten the type attribute if
you're going to use it more than once.

public interface CacheController {
  public void flush(CacheModel cacheModel);
  public Object getObject(CacheModel cacheModel, Object key);
  public Object removeObject(CacheModel cacheModel, Object key);
  public void putObject(CacheModel cacheModel, Object key, Object object);
  public void setProperties(Properties props);

Hope that helps,


Comment on this post