Package lepl :: Package lexer :: Module matchers :: Class Token
[hide private]
[frames] | no frames]

Class Token

source code


A token with a user-specified regexp.
Instance Methods [hide private]
 
__init__(self, regexp, content=None, id_=None, alphabet=None, complete=True, compiled=False)
Define a token that will be generated by the lexer.
source code

Inherited from BaseToken: __call__, __repr__, __str__, compile

Inherited from BaseToken (private): _match

Class Methods [hide private]

Inherited from BaseToken: reset_ids

Method Details [hide private]

__init__(self, regexp, content=None, id_=None, alphabet=None, complete=True, compiled=False)
(Constructor)

source code 

Define a token that will be generated by the lexer.

regexp is the regular expression that the lexer will use to generate
appropriate tokens.

content is the optional matcher that will be invoked on the value
of the token.  It is usually set via (), which clones this instance
so that the same token can be used more than once.

id_ is an optional unique identifier that will be given an integer
value if left empty.

alphabet is the alphabet associated with the regexp.  It should be
set by the lexer rewiter, so that all instances share the same
value (it appears in the constructor so that Tokens can be cloned).

complete indicates whether any sub-matcher must completely exhaust
the contents when matching.  It can be over-ridden for a particular
sub-matcher via __call__().

compiled should only be used internally.  It is a flag indicating
that the Token has been processed by the rewriter (see below).

A Token must be "compiled" --- this completes the configuration
using a given alphabet and is done by the lexer_rewriter.  Care is taken
to allow a Token to be cloned before or after compilation.

Overrides: BaseToken.__init__