| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Crypto AG DID work with NSA / GCHQ; UNUMS (Universal Number Format); MOOCs (Massive Open Online Courses); Interesting Looking Game; Euler's Theorem for Polynomials; Weeks 3-6; Reddit Comment; Differential Cryptanalysis For Dummies; Japanese Graphic Design; Books To Be Re-Read; And Today I Learned Bugs Need Clear Examples; Factoring a 67 bit prime in your head; Islamic Geometric Art; Useful Julia Backtraces from Tasks; Nothing, however, is lost with less discomfort than that which, when lost, cannot be missed; Article on Didion; Cost of Living by City; British Slavery; Derrida on Metaphor; African SciFi; Traits in Julia; Alternative Japanese Lit; Pulic Key as Address (Snow); Why Information Grows; The Blindness Of The Chilean Elite; Some Victoriagate Links; This Is Why I Left StackOverflow; New TLS Implementation; Maths for Physicists; How I Am 8; 1000 Word Philosophy; Cyberpunk Reading List; Detailed Discussion of Message Dispatch in ParserCombinator Library for Julia; FizzBuzz in Julia w Dependent Types; kokko - Design Shop in Osaka; Summary of Greece, Currently; LLVM and GPUs; See Also; Schoolgirl Groyps (Maths); Japanese Lit; Another Example - Modular Arithmetic; Music from United; Python 2 and 3 compatible alternative.; Read Agatha Christie for the Plot; A Constructive Look at TempleOS; Music Thread w Many Recommendations; Fixed Version; A Useful Julia Macro To Define Equality And Hash; k3b cdrom access, OpenSuse 13.1; Week 2; From outside, the UK looks less than stellar; Huge Fonts in VirtualBox; Keen - Complex Emergencies; The Fallen of World War II; Some Spanish Fiction; Calling C From Fortran 95; Bjork DJ Set; Z3 Example With Python; Week 1; Useful Guide To Starting With IJulia; UK Election + Media; Review: Reinventing Organizations; Inline Assembly With Julia / LLVM; Against the definition of types; Dumb Crypto Paper; The Search For Quasi-Periodicity...; Is There An Alternative To Processing?; CARDIAC (CARDboard Illustrative Aid to Computation); The Bolivian Case Against Chile At The Hague; Clear, Cogent Economic Arguments For Immigration; A Program To Say If I Am Working; Decent Cards For Ill People; New Photo; Luksic And Barrick Gold; President Bachelet's Speech; Baltimore Primer; libxml2 Parsing Stream; configure.ac Recipe For Library Path; The Davalos Affair For Idiots; Not The Onion: Google Fireside Chat w Kissinger; Bicycle Wheels, Inertia, and Energy; Another Tax Fraud; Google's Borg; A Verion That Redirects To Local HTTP Server; Spanish Accents For Idiots; Aluminium Cans; Advice on Spray Painting; Female View of Online Chat From a Male; UX Reading List; S4 Subgroups - Geometric Interpretation; Fucking Email; The SQM Affair For Idiots; Using Kolmogorov Complexity; Oblique Strategies in bash; Curses Tools; Markov Chain Monte Carlo Without all the Bullshit; Email Para Matias Godoy Mercado; The Penta Affair For Idiots; Example Code To Create numpy Array in C; Good Article on Bias in Graphic Design (NYTimes); Do You Backup github?

© 2006-2015 Andrew Cooke (site) / post authors (content).

Useful Java Proeprties Parsing Idiom

From: "andrew cooke" <andrew@...>

Date: Sun, 19 Mar 2006 17:07:48 -0400 (CLT)

Java isn't the easiest language to write simple parsers in.  Often that
doesn't matter because you can use XML, but at work I had a requirement
for "easy configuration" which, I think, meant "enough of your damn Spring
verbosity, it's gotta be text".

I have the impression (although now I can't think of any examples... ah, I
think it's used by Tiles configuration, at least) that it's quite common
to use Java properties file for config.  They inherit usefully, support
comments, and are "easy to read".

More exactly - since Properties files are of course used for config - the
following idiom seems to be fairly common:

  # two column table with string values

  table1.sql.table=qualified.name
  table1.sql.column.column1.header=HEADER1
  table1.sql.column.column1.type=varchar
  table1.sql.column.column2.header=HEADER2
  table1.sql.column.column2.type=varchar

  # junk loader and image

  table1.java.loader=java.lang.Object
  table1.java.image=java.lang.Object

And I hope it's intuitively obvious that's some kind of SQL related
description of tables.  You don't need to know more than that about the
application (or I'd have to kill you).

But how to parse that?

First, what does it parse to?  There's a fairly obvious tree model, which
I had designed earlier (and which matches the above nicely):

  trees
   +- table1
       +- modelName (=table1)
       +- table (=qualified.name)
       +- loader (=java.lang.Object)
       +- image (=java.lang.Object)
       +- columns
           +- column1
           |   +- modelName (=column1)
           :   +- type (=varchar)
               +- header (=HEADER1)

The nice thing to note is that just because of the way the names in the
properties file are built, you have to define a minimal path through the
tree for any value.  So you can build the tree using incomplete nodes "as
you go" and don't need to force the user to supply values in any order.

However that means that you can end up with incomplete nodes, so you need
a recursive verify() operation that checks everything is defined.

It also simplifies things to have
  Node addNode(String name)
methods that add the node if it's mising and return the named node.  This
helps you quickly step down through the tree as you handle any particular
specification.

One thing I didn't do, which I probably should have done, was tokenize the
name values.  instead I repepatedly parse them, which is inefficient (but
allows the separator value to be used if context requires it).

However, the neatest trick was the use of enums.  Many of the name tokens
are constants (sql, column, header, image, etc).  So I had code like:

  private static enum Domain {SQL, JAVA, META};
  private static enum SqlSpec {COLUMN, TABLE, OTHER};

And here's how the token is converted to an enum:

  private SqlSpec getSqlSpec(String line) throws ModelException {
    return (SqlSpec)getEnum(line, 2, SqlSpec.values(), SqlSpec.OTHER);
  }

  private Enum getEnum(String line, int chunk, Enum[] values, Enum deflt)
  throws Exception {
    String name = getChunk(chunk, line);
    for (Enum value: values) {
      if (value.name().equalsIgnoreCase(name)) return value;
    }
    return deflt;
  }

where getChunk gets the numbered token (as I said, that bit is not so cool).

So then the parse logic becomes trivial (this is already one select down):

  private void parseSql(String line, String value, Table table) {
    switch(getSqlSpec(line)) {
    case COLUMN:
      String name = getChunk(3, line, line);
      Column column = table.addColumn(name);
      parseColumn(line, value, column); // further descent
      break;
    case TABLE:
      table.setDatabaseName(value);
      break;
    case OTHER:
     throw new RuntimeException("Unsupported specification: " + line);
    }
  }

A summary:
 - allow any order in specification
 - construct paths with incomplete values
 - do a final recursive verify (verification in tree node classes)
 - Node addNode(String name) simplifies rapid traversal
 - use enums for sets for constant values
 - use nested selects with the enums

And now I'm going to change it to use StringTokenizer and avoid that ugly
getChunk...

Andrew

Now With Tokens (Extra Crunch)

From: "andrew cooke" <andrew@...>

Date: Sun, 19 Mar 2006 17:24:41 -0400 (CLT)

It was easy to rewrite with tokens.  The code is about the same - slightly
messier, I think, but none of the explicit depths are there (in the calls
to getChunk).

Maybe that doesn't sound like much, but of the four errors I found via my
unit tests, three were related to those values.  :o/

Still, I guess being wise after the event is better than never.

Refinements and Comments on the Properties Parser

From: "andrew cooke" <andrew@...>

Date: Thu, 23 Mar 2006 19:36:28 -0400 (CLT)

1 - Not Properties

It had to happen - at some point I change the format so that the order
mattered.  And then found out that Java.util.Properties doesn't preserve
ordering in the files.  So I neded up writing my own (which is not a lot
of work and allowed me to keep track of source/line for better error
reporting).

2 - Theory

A colleague here was using a simple parser package and I relaised I'd just
rushed in without thinking about theory, as usual.  A little reflection
shows that what I'm doing here is basically recursive descent (although in
partice there's little recursion) with context-dependent lexing.

That made me look again at the lexer part, which was kind-of mixed in with
the rules/productions.  I ended up pulling it out into a separate (inner)
class:

  protected static class Lexer<E extends Enum<E>>  {

    private E[] values;

    public Lexer(E[] values) {
      this.values = values;
    }

    public E lex(String name, Line line) {
      for (E value: values) {
        if (value.name().toLowerCase().startsWith(name.toLowerCase()) {
          return value;
        }
      }
      throw // error can include name and list of expected values
    }
  }

To use this, define an enum and subclass the lexer:

  private static enum TableSpec {COLUMN, DBNAME, BINDER, IMAGE, ERROR}
  private static Lexer<TableSpec> tableLexer =
    new Lexer<TableSpec>(TableSpec.values());

This leaves the rules/production in a simple switch:

  private void parseTable(StringTokenizer tokens, Node node, ...) {
    switch(tableLexer.lex(tokens)) {
    case COLUMN:
      Node child = node.addChild(...); // accumulate
      parseColumn(tokens, child, ..); // recursion
      break;
    case DBNAME:
      // ...
    case BINDER:
      // ...
    }
  }

3 - Accumulate the Parse Tree

Using an accumulator approach to constructing the parse tree simplifies
the productions (there's no backtracking so no need to worry about
destructive state).  So each node has an "addChild" method, which returns
either a known value (if traversing an already existing path), or a new
value.

4 - Embedded Names

There were two kinds of nodes in my tree.  One kind had a fixed set of
children.  Parsing the children required a lexer as described above. 
Typically this corresponded to selecting an option (for example, a data
type).

Other nodes could take "any" value and the String token is used directly
(the StringTokenizer splits on "." - see earlier discussion).  Typically
these nodes are names and the parent includes a map from name to node.

5 - Damn Enums

I still can't work out a way to generate the full set of enums
automatically (see the Lexer class above, which takes an array as an
argument).  I tried both using the generic type parameter and passing the
Enum itself with no success (ie I'd like new Lexer<TableSpec>() or
Lexer<TableSpec>(TableSpec)
to be possible).  I guess the latter should be TableSpec.class - somehow I
seem to be confusing classes and objects.

Comment on this post