Home       Trees       Indices       Help   
Package lepl :: Package lexer :: Module matchers :: Class BaseToken
[hide private]
[frames] | no frames]

Class BaseToken

source code

Introduce a token that will be recognised by the lexer. A Token instance can be specialised to match particular contents by calling as a function.

This is a base class that provides all the functionality, but doesn't set the regexp attribute. This allows subclasses to provide a fixed value, while Token uses the constructor.

Instance Methods [hide private]
 
__init__(self, content=None, id_=None, alphabet=None, complete=True, compiled=False)
Define a token that will be generated by the lexer.		 source code
 
compile(self, alphabet=None)
Convert the regexp if necessary.		 source code
 
__call__(self, content, complete=None)
If complete is specified as True of False it overrides the value set in the constructor.		 source code
 
_match(self, stream)
On matching we first assert that the token type is correct and then delegate to the content.		 source code
 
__str__(self) source code
 
__repr__(self) source code
Class Methods [hide private]
 
reset_ids(cls)
Reset the ID counter.		 source code
Static Methods [hide private]
 
__to_regexp(regexp, alphabet)
The regexp may be a matcher; if so we try to convert it to a regular expression and extract the equivalent text.		 source code
Class Variables [hide private]
  __count = 0
Method Details [hide private]

__init__(self, content=None, id_=None, alphabet=None, complete=True, compiled=False)
(Constructor)

source code  

Define a token that will be generated by the lexer.

content is the optional matcher that will be invoked on the value
of the token.  It is usually set via (), which clones this instance
so that the same token can be used more than once.

id_ is an optional unique identifier that will be given an integer
value if left empty.

alphabet is the alphabet associated with the regexp.  It should be
set by the lexer rewiter, so that all instances share the same
value (it appears in the constructor so that Tokens can be cloned).

complete indicates whether any sub-matcher must completely exhaust
the contents when matching.  It can be over-ridden for a particular
sub-matcher via __call__().

compiled should only be used internally.  It is a flag indicating
that the Token has been processed by the rewriter (see below).

A Token must be "compiled" --- this completes the configuration
using a given alphabet and is done by the lexer_rewriter.  Care is 
taken to allow a Token to be cloned before or after compilation.

__call__(self, content, complete=None)
(Call operator)

source code 
If complete is specified as True of False it overrides the value set in the constructor. If True the content matcher must complete match the Token contents.

_match(self, stream)

source code 
On matching we first assert that the token type is correct and then delegate to the content.
Decorators:
  • @tagged

reset_ids(cls)
Class Method

source code 
Reset the ID counter. This should not be needed in normal use.

   Home       Trees       Indices       Help   
Generated by Epydoc 3.0.1 on Sat Jun 9 21:50:53 2012 http://epydoc.sourceforge.net