| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Calling C From Fortran 95; Bjork DJ Set; Z3 Example With Python; Week 1; Useful Guide To Starting With IJulia; UK Election + Media; Review: Reinventing Organizations; Inline Assembly With Julia / LLVM; Against the definition of types; Dumb Crypto Paper; The Search For Quasi-Periodicity...; Is There An Alternative To Processing?; CARDIAC (CARDboard Illustrative Aid to Computation); The Bolivian Case Against Chile At The Hague; Clear, Cogent Economic Arguments For Immigration; A Program To Say If I Am Working; Decent Cards For Ill People; New Photo; Luksic And Barrick Gold; President Bachelet's Speech; Baltimore Primer; libxml2 Parsing Stream; configure.ac Recipe For Library Path; The Davalos Affair For Idiots; Not The Onion: Google Fireside Chat w Kissinger; Bicycle Wheels, Inertia, and Energy; Another Tax Fraud; Google's Borg; A Verion That Redirects To Local HTTP Server; Spanish Accents For Idiots; Aluminium Cans; Advice on Spray Painting; Female View of Online Chat From a Male; UX Reading List; S4 Subgroups - Geometric Interpretation; Fucking Email; The SQM Affair For Idiots; Using Kolmogorov Complexity; Oblique Strategies in bash; Curses Tools; Markov Chain Monte Carlo Without all the Bullshit; Email Para Matias Godoy Mercado; The Penta Affair For Idiots; Example Code To Create numpy Array in C; Good Article on Bias in Graphic Design (NYTimes); Do You Backup github?; Data Mining Books; SimpleDateFormat should be synchronized; British Words; Chinese Govt Intercepts External Web To DDOS github; Numbering Permutations; Teenage Engineering - Low Price Synths; GCHQ Can Do Whatever It Wants; Dublinesque; A Cryptographic SAT Solver; Security Challenges; Word Lists for Crosswords; 3D Printing and Speaker Design; Searchable Snowden Archive; XCode Backdoored; Derived Apps Have Malware (CIA); Rowhammer - Hacking Software Via Hardware (DRAM) Bugs; Immutable SQL Database (Kinda); Tor GPS Tracker; That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route

© 2006-2015 Andrew Cooke (site) / post authors (content).

Lepl 4 Preview - Simpler, Faster, Easier

From: andrew cooke <andrew@...>

Date: Thu, 11 Mar 2010 21:55:49 -0300

Here's a draft of the page from the Lepl docs that describes the new features.
This is *not* released so, at the tie of writing, it does *not* describe the
current library, or web site (but if you're impatient, everything you read
here is available for download from the source repo!).



Lepl 4 - Simpler, Faster, Easier
================================

I've made Lepl simpler to use.  For example, if a parser fails to match the
input, you get an exception with the location of the problem.  If that's not
what you want, it can be disabled by calling `.no_full_match()` (configuration
got simpler too!).

Another example: it's easier to add new matchers.  Before, you needed to
subclass a complex class.  Now, you can add a decorator to a simple function.

Even debugging is simpler.  If you want to understand what the parser is
doing, add `with TrackVariables()` and the progress of the match will be
printed to your screen.  The display includes the variable names that you used
in the code, so it's easy to understand.

Often when software is made simpler to use, it becomes slower.  The reverse is
true for Lepl - the new, simpler, approach supports new optimisations and
makes fixing bugs easier.  In my tests, parsers using the default
configuration are up to 10 times faster.

Below I'll explain all these new features in much more detail, but if you want
to get started with Lepl now, installation instructions are on the (new,
simpler) front page.


A Simpler API
-------------

Configuration
~~~~~~~~~~~~

Matches are now configured via methods on the `.config` attribute.  There's no
need to hunt round the documentation looking for rewriters -- everything is
right at your fingertips::

   >>> matcher = Any()
   >>> dir(matcher.config)
   [...]
   >>> help(matcher.config.left_memoize)
   
   Help on method left_memoize in module lepl.core.config:
   
   left_memoize(self) method of lepl.core.config.ConfigBuilder instance
       Add memoization that can detect and stabilise left-recursion.  This
       makes the parser more robust (so it can handle more grammars) but
       also significantly slower.

Each configuration option has two methods --- one to turn it on, and one to
turn it off.  These changes are relative to the default configuration [TODO
reference] unless you first call `.clear()` (which removes all options).

So, for example::

  >>> matcher.config.no_lexer()

removes lexer support from the default configuration, while::

  >>> matcher.clear().lexer()

gives a configuration that *only* has lexer support.


Full Match
~~~~~~~~~

Often, particularly with a simple parser, you expect all the input to be
matched.  If it isn't, something went wrong, and you'd like to know where.

In Lepl 4 you get all that by default::

  >>> matcher = Any()[5]
  
  >>> try:
  >>>     matcher.parse('1234567')
  >>> except FullMatchException as e:
  >>>     print(str(e))
  The match failed at '67'.

And if you use a more specific parse method, you get a more detailed error::
   
  >>> try:
  >>>     matcher.parse_string('1234567')
  >>> except FullMatchException as e:
  >>>     print(str(e))
  The match failed at '67',
  Line 1, character 5 of str: '1234567'.

Of course, you can disable this with `.config.no_full_match()`.

For more details, see the manual [TODO - reference].


Multiple Matches, Parsers
~~~~~~~~~~~~~~~~~~~~~~~~

The new `.parse_all()` method (and related `.parse_string_all()`, etc) returns
a generator of all possible matches.  This is similar to the old `.match()`
method (which still exists), but without the remaining streams (which were
usually not interesting).  If you need multiple matches you'll probably find
that `.parse_all()` simplifies your code.

Also, parsers are now cached (this isn't strictly new - it was also present in
later Lepl 3 versions).  This means that you can call `.parse()` repeatedly
without worrying about wasting time re-compiling the parser.

Cached parsers and configuration interact like you would expect --- changing
the configuration clears the cache so that a new parser is compiled with the
new settings.  If you want to keep a copy of the parser with the old settings
(useful in tests) then try `.get_parser()`.


Upgrading from Lepl 3
~~~~~~~~~~~~~~~~~~~~

Lepl 4.0 is a major release, which means that it contains changes to
often-used methods.  If you have code from a previous version the changes
described here are important, because they will probably cause your program to
fail.  The good news is that the parts of the API with most changes are those
that are called only once (configuration, creating the parser, etc).  So
updating your code should be relatively easy.  In particular, the way that the
grammar is specified is unchanged.


Easier to Extend
----------------

Roll Your Own Matcher
~~~~~~~~~~~~~~~~~~~~

Adding a new matcher to Lepl is now as easy as writing a function::

  >>> @function_matcher
  >>> def Capital(support, stream):
  ...    '''A matcher for capital letters.'''
  ...    if stream[0] in ascii_uppercase:
  ...        return ([stream[0]], stream[1:])
  ...
  >>> Capital.config.no_full_match()
  >>> Capital.parse('ABC')
  ['A']

If the matcher supports multiple results then it should `yield` them::

  >>> @sequence_matcher
  ... def Digit(support, stream):
  ...     '''Provide all possible telephone keypresses.'''
  ...     digits = {'1': '',     '2': 'abc',  '3': 'def',
  ...               '4': 'ghi',  '5': 'jkl',  '6': 'mno',
  ...               '7': 'pqrs', '8': 'tuv',  '9': 'wxyz',
  ...               '0': ''}
  ...     if stream:
  ...         digit, tail = stream[0], stream[1:]
  ...         yield ([digit], tail)
  ...         if digit in digits:
  ...             for letter in digits[digit]:
  ...                 yield ([letter], tail)
  ...
  >>> list(Digit()[3, ...].parse_all('123'))
  [['123'], ['12d'], ['12e'], ['12f'], ['1a3'], ['1ad'], ['1ae'], ['1af'], 
  ['1b3'], ['1bd'], ['1be'], ['1bf'], ['1c3'], ['1cd'], ['1ce'], ['1cf']]

Note how these matchers inherit the full functionality of Lepl!

For more information, including support for matchers that process other
matchers, or be configured in the grammar, see [TODO].


General Transformations
~~~~~~~~~~~~~~~~~~~~~~

Lepl has always supported functions that transform results, but the underlying
implementation is now signifcantly more powerful.  For example, a function may
add alternative matches, or abort the matching early.

This functionality is unlikely to be used in grammars, but will make adding
cool new features easier.


Easier Debugging
----------------

The `Trace()` functionality in Lepl has never been easy to understand, for two
reasons.  First, it tracks *every* matcher.  Second, it's unclear which
matcher corresponds to which part of the grammar.

Normally, when we debug a program, things are simpler because we can see the
*variables*.  So I have added that to Lepl.  The implementation has some rough
corners, because it uses parts of Python that were not intended to be used in
this way, but I think you'll agree that the result is worth the effort.

Here's an example.  The variables that will be displayed must be defined
inside `with TrackVariables()`::

  >>> with TrackVariables():
  ...     word = ~Lookahead('OR') & Word()
  ...     phrase = String()
  ...     with DroppedSpace():
  ...         text = (phrase | word)[1:] > list
  ...         query = text[:, Drop('OR')]
  ...
  >>> query.parse('spicy meatballs OR "el bulli restaurant"')
        phrase failed                             stream = 'spicy meatballs
  OR...
          word = ['spicy']                        stream = ' meatballs OR "el
  ...
        phrase failed                             stream = 'meatballs OR "el
  b...
          word = ['meatballs']                    stream = ' OR "el bulli
  rest...
        phrase failed                             stream = 'OR "el bulli
  resta...
          word failed                             stream = 'OR "el bulli
  resta...
        phrase failed                             stream = ' OR "el bulli
  rest...
          word failed                             stream = ' OR "el bulli
  rest...
          text = [['spicy', 'meatballs']]         stream = ' OR "el bulli
  rest...
        phrase = ['el bulli restaurant']          stream = ''
        phrase failed                             stream = ''
          word failed                             stream = ''
          text = [['el bulli restaurant']]        stream = ''
  [['spicy', 'meatballs'], ['el bulli restaurant']]



Faster Parsers
--------------

Faster Defaults
~~~~~~~~~~~~~~

I spent some time profiling, experimenting with different configurations, and
have tweaked the default settings so that, on average, parsers are faster.  In
particular, memoisation is used only to detect left--recursive loops (if you
do want full memoisation you can still configure it, of course, with
`.config.auto_memoize(full=True)`).


No Trampolining
~~~~~~~~~~~~~~

Lepl is unique (I believe) in using trampoling and co-routines to implement
the recursive descent.  This has several advantages, but introduces some
overhead.

I have measured the overhead, and it's surprisingly small, but even so it
seems silly to have it when it's not needed.  But the problem has always been:
when is it not needed?  The ability to define matchers via functions,
described above, finally gave an answer to that question.

Matchers that are defined as functions are simpler than a completely general
matcher.  So Lepl exploits this to remove trampolining when they are used.
And, of course, matchers provided by Lepl are implemented this way when
possible.

The end result is that trampoling is removed when the grammar is unlikely to
need it.  If you disagree you add it back through the configuration
(`.config.no_direct_eval()`).


Better Memoisation
~~~~~~~~~~~~~~~~~~

Sometimes memoisation is a *big* win.  It's not enabled by default, so you
still need to experiment to find out when to use it.  But until now it had a
stupid bug that made it less likely to work.  That bug is now fixed, so when
you need memoisation, it will be there for you.

Comment on this post