Token

The Token class is the class of tokens which are found by a TokenMarker in a text.

Tokens are created by the TokenMarker associated with the syntax when tokenizing a text for a particular syntax.

Characteristics of a Token

A Token has the following fields:

• The Token.id is the type of the Token. For example the Token.METHOD is a Token which represents a method declaration
• The Token.length is the length of the String of the Token

Types of a Token

A token can have the following types^[1]

See Token.id

:

• Token.NULL: the default token with no syntax
• Token.COMMENT1: a token for a comment
• Token.COMMENT2: a token for another type of comment
• Token.LITERAL1: a token for a string literal (for examplme, C syntax uses this to mark "..." literals)
• Token.LITERAL2: a token for an object literal (for examplme, Java syntax uses this to mark true, false, etc)
• Token.LABEL: a token for a label
• Token.KEYWORD1: a token for a keyword. This should be used for general language constructs
• Token.KEYWORD2: a token for a keyword. This should be used for preprocessor commands, or variables
• Token.KEYWORD3: a token for a keyword. This should be used for data types
• Token.OPERATOR: a token for an operator (for example + or -)
• Token.METHOD: a token for a method
• Token.END: marks the end of the tokens list
• Token.INVALID: used for an invalid or incomplete tokens