| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Payroll Service Quotes for Cat Soft LLC; (Beating Dead Horse) StackOverflow; Current State of Justice in China; Now Is Cat Soft LLC's Chance To Save Up To 32% On Mail; Axiom of Determinacy; Ewww; Fee Chaos Book; Course on Differential Geometry; Increase Efficiency with GPS Vehicle Tracking for Cat Soft LLC; Okay, but...; Sparse Matrices, Deep Learning; Sounds Bad; Applebaum Rape; Tomato Chutney v4; Have to add...; Culturally Liberal and Nothing More; Weird Finite / Infinite Result; Your diamond is a beaten up mess; Maths Books; Good Bike Route from Providencia / Las Condes to Panul\; Iain Pears (Author of Complex Plots); Plum Jam; Excellent; More Recently; For a moment I forgot StackOverflow sucked; A Few Weeks On...; Chilean Book Recommendations; How To Write Shared Libraries; Jenny Erpenbeck (Author); Dijkstra, Coins, Tables; Python libraries error on OpenSuse; Deserving Trump; And Smugness; McCloskey Economics Trilogy; cmocka - Mocks for C; Concept Creep (Americans); Futhark - OpenCL Language; Moved / Gone; Fan and USB issues; Burgers in Santiago; The Origin of Icosahedral Symmetry in Viruses; autoenum on PyPI; Jars Explains; Tomato Chutney v3; REST; US Elections and Gender: 24 Point Swing; PPPoE on OpenSuse Leap 42.1; SuperMicro X10SDV-TLN4F/F with Opensuse Leap 42.1; Big Data AI Could Be Very Bad Indeed....; Cornering; Postcapitalism (Paul Mason); Black Science Fiction; Git is not a CDN; Mining of Massive Data Sets; Rachel Kaadzi Ghansah; How great republics meet their end; Raspberry, Strawberry and Banana Jam; Interesting Dead Areas of Math; Later Taste; For Sale; Death By Bean; It's Good!; Tomato Chutney v2; Time ATAC MX 2 Pedals - First Impressions; Online Chilean Crafts; Intellectual Variety; Taste + Texture; Time Invariance and Gauge Symmetry; Jodorowsky; Tomato Chutney; Analysis of Support for Trump; Indian SF; TP-Link TL-WR841N DNS TCP Bug; TP-Link TL-WR841N as Wireless Bridge; Sending Email On Time; Maybe run a command; Sterile Neutrinos; Strawberry and Banana Jam; The Best Of All Possible Worlds; Kenzaburo Oe: The Changeling; Peach Jam; Taste Test; Strawberry and Raspberry Jam; flac to mp3 on OpenSuse 42.1; Also, Sebald; Kenzaburo Oe Interview; Otake (Kitani Minoru) move Black 121; Is free speech in British universities under threat?; I am actually good at computers; Was This Mansplaining?; WebFaction / LetsEncrypt / General Disappointment; Sensible Philosophy of Science; George Ellis; Misplaced Intuition and Online Communities; More Reading About Japan; Visibilty / Public Comments / Domestic Violence; Ferias de Santiago; More (Clearly Deliberate); Deleted Obit Post; And then a 50 yo male posts this...; We Have Both Kinds Of Contributors

© 2006-2015 Andrew Cooke (site) / post authors (content).

Useful Java Proeprties Parsing Idiom

From: "andrew cooke" <andrew@...>

Date: Sun, 19 Mar 2006 17:07:48 -0400 (CLT)

Java isn't the easiest language to write simple parsers in.  Often that
doesn't matter because you can use XML, but at work I had a requirement
for "easy configuration" which, I think, meant "enough of your damn Spring
verbosity, it's gotta be text".

I have the impression (although now I can't think of any examples... ah, I
think it's used by Tiles configuration, at least) that it's quite common
to use Java properties file for config.  They inherit usefully, support
comments, and are "easy to read".

More exactly - since Properties files are of course used for config - the
following idiom seems to be fairly common:

  # two column table with string values

  table1.sql.table=qualified.name
  table1.sql.column.column1.header=HEADER1
  table1.sql.column.column1.type=varchar
  table1.sql.column.column2.header=HEADER2
  table1.sql.column.column2.type=varchar

  # junk loader and image

  table1.java.loader=java.lang.Object
  table1.java.image=java.lang.Object

And I hope it's intuitively obvious that's some kind of SQL related
description of tables.  You don't need to know more than that about the
application (or I'd have to kill you).

But how to parse that?

First, what does it parse to?  There's a fairly obvious tree model, which
I had designed earlier (and which matches the above nicely):

  trees
   +- table1
       +- modelName (=table1)
       +- table (=qualified.name)
       +- loader (=java.lang.Object)
       +- image (=java.lang.Object)
       +- columns
           +- column1
           |   +- modelName (=column1)
           :   +- type (=varchar)
               +- header (=HEADER1)

The nice thing to note is that just because of the way the names in the
properties file are built, you have to define a minimal path through the
tree for any value.  So you can build the tree using incomplete nodes "as
you go" and don't need to force the user to supply values in any order.

However that means that you can end up with incomplete nodes, so you need
a recursive verify() operation that checks everything is defined.

It also simplifies things to have
  Node addNode(String name)
methods that add the node if it's mising and return the named node.  This
helps you quickly step down through the tree as you handle any particular
specification.

One thing I didn't do, which I probably should have done, was tokenize the
name values.  instead I repepatedly parse them, which is inefficient (but
allows the separator value to be used if context requires it).

However, the neatest trick was the use of enums.  Many of the name tokens
are constants (sql, column, header, image, etc).  So I had code like:

  private static enum Domain {SQL, JAVA, META};
  private static enum SqlSpec {COLUMN, TABLE, OTHER};

And here's how the token is converted to an enum:

  private SqlSpec getSqlSpec(String line) throws ModelException {
    return (SqlSpec)getEnum(line, 2, SqlSpec.values(), SqlSpec.OTHER);
  }

  private Enum getEnum(String line, int chunk, Enum[] values, Enum deflt)
  throws Exception {
    String name = getChunk(chunk, line);
    for (Enum value: values) {
      if (value.name().equalsIgnoreCase(name)) return value;
    }
    return deflt;
  }

where getChunk gets the numbered token (as I said, that bit is not so cool).

So then the parse logic becomes trivial (this is already one select down):

  private void parseSql(String line, String value, Table table) {
    switch(getSqlSpec(line)) {
    case COLUMN:
      String name = getChunk(3, line, line);
      Column column = table.addColumn(name);
      parseColumn(line, value, column); // further descent
      break;
    case TABLE:
      table.setDatabaseName(value);
      break;
    case OTHER:
     throw new RuntimeException("Unsupported specification: " + line);
    }
  }

A summary:
 - allow any order in specification
 - construct paths with incomplete values
 - do a final recursive verify (verification in tree node classes)
 - Node addNode(String name) simplifies rapid traversal
 - use enums for sets for constant values
 - use nested selects with the enums

And now I'm going to change it to use StringTokenizer and avoid that ugly
getChunk...

Andrew

Now With Tokens (Extra Crunch)

From: "andrew cooke" <andrew@...>

Date: Sun, 19 Mar 2006 17:24:41 -0400 (CLT)

It was easy to rewrite with tokens.  The code is about the same - slightly
messier, I think, but none of the explicit depths are there (in the calls
to getChunk).

Maybe that doesn't sound like much, but of the four errors I found via my
unit tests, three were related to those values.  :o/

Still, I guess being wise after the event is better than never.

Refinements and Comments on the Properties Parser

From: "andrew cooke" <andrew@...>

Date: Thu, 23 Mar 2006 19:36:28 -0400 (CLT)

1 - Not Properties

It had to happen - at some point I change the format so that the order
mattered.  And then found out that Java.util.Properties doesn't preserve
ordering in the files.  So I neded up writing my own (which is not a lot
of work and allowed me to keep track of source/line for better error
reporting).

2 - Theory

A colleague here was using a simple parser package and I relaised I'd just
rushed in without thinking about theory, as usual.  A little reflection
shows that what I'm doing here is basically recursive descent (although in
partice there's little recursion) with context-dependent lexing.

That made me look again at the lexer part, which was kind-of mixed in with
the rules/productions.  I ended up pulling it out into a separate (inner)
class:

  protected static class Lexer<E extends Enum<E>>  {

    private E[] values;

    public Lexer(E[] values) {
      this.values = values;
    }

    public E lex(String name, Line line) {
      for (E value: values) {
        if (value.name().toLowerCase().startsWith(name.toLowerCase()) {
          return value;
        }
      }
      throw // error can include name and list of expected values
    }
  }

To use this, define an enum and subclass the lexer:

  private static enum TableSpec {COLUMN, DBNAME, BINDER, IMAGE, ERROR}
  private static Lexer<TableSpec> tableLexer =
    new Lexer<TableSpec>(TableSpec.values());

This leaves the rules/production in a simple switch:

  private void parseTable(StringTokenizer tokens, Node node, ...) {
    switch(tableLexer.lex(tokens)) {
    case COLUMN:
      Node child = node.addChild(...); // accumulate
      parseColumn(tokens, child, ..); // recursion
      break;
    case DBNAME:
      // ...
    case BINDER:
      // ...
    }
  }

3 - Accumulate the Parse Tree

Using an accumulator approach to constructing the parse tree simplifies
the productions (there's no backtracking so no need to worry about
destructive state).  So each node has an "addChild" method, which returns
either a known value (if traversing an already existing path), or a new
value.

4 - Embedded Names

There were two kinds of nodes in my tree.  One kind had a fixed set of
children.  Parsing the children required a lexer as described above. 
Typically this corresponded to selecting an option (for example, a data
type).

Other nodes could take "any" value and the String token is used directly
(the StringTokenizer splits on "." - see earlier discussion).  Typically
these nodes are names and the parent includes a map from name to node.

5 - Damn Enums

I still can't work out a way to generate the full set of enums
automatically (see the Lexer class above, which takes an array as an
argument).  I tried both using the generic type parameter and passing the
Enum itself with no success (ie I'd like new Lexer<TableSpec>() or
Lexer<TableSpec>(TableSpec)
to be possible).  I guess the latter should be TableSpec.class - somehow I
seem to be confusing classes and objects.

Comment on this post