Token

The Token class is the class of tokens which are found by a TokenMarker in a text.

Tokens are created by the TokenMarker associated with the syntax when tokenizing a text for a particular syntax.

Characteristics of a Token

A Token has the following fields:

• The Token.id is the type of the Token. For example the Token.METHOD is a Token which represents a method declaration
• The Token.length is the length of the String of the Token