Home
Categories
Dictionary
Download
Project Details
Changes Log
License

TokenizedDocument

    1 Creating a TokenizedDocument
    2 Applying the syntax to get the tokens
    3 Iterating through the tokens
    4 Example
    5 See also

The TokenizedDocument class allows to iterate on the Tokens in a SyntaxDocument.

Creating a TokenizedDocument

It is possible to create a TokenizedDocument from a SyntaxDocument by the SyntaxDocument.tokenize() method.

Creating the TokenizedDocument also applies the syntax, which means that it is not necessary to call the TokenizedDocument.applySyntax() method.

Applying the syntax to get the tokens

The TokenizedDocument.applySyntax() populates the list of Tokens of the associated document. The TokenizedDocument.getTokens() returns the ordered Map of PosTokens of tokens in the TokenizedDocument.

Iterating through the tokens

The following methods allow to return an iterator on the PosTokens before or after a position in the document:

The TokenizedDocument.iterateBefore(int, int) iterates on tokens before the specified position, and of a specified id. The iterator is applied backwards from the closest token to the position to the first token of the document
The TokenizedDocument.iterateAfter(int, int) iterates on tokens before the specified position, and of a specified id. The iterator is applied forwards from the closest token to the position to the last token of the document

If the specified id is -1, the iterator will return all Tokens, regardless of their types.

Example

The following code allows to get a TokenizedDocument, and iterate through Tokens of the Token.KEYWORD1 type, before a position:

     JEditor editor = new JEditor();
     editor.setText(url);
     editor.setTokenMarker(new XMLTokenMarker());

     SyntaxDocument syntaxdoc = editor.getSyntaxDocument();
     TokenizedDocument tokDoc = syntaxdoc.tokenize();  
       
     Iterator<PosToken> it = tokDoc.iterateBefore(<the index>, Token.KEYWORD1);