TokenMarker (jEditor API)

java.lang.Object
- org.jeditor.scripts.base.TokenMarker

Direct Known Subclasses:

BasicTokenMarker, BatchFileTokenMarker, CSharpTokenMarker, CSSTokenMarker, CsvTokenMarker, CTokenMarker, EiffelTokenMarker, FortranTokenMarker, HTMLTokenMarker, MakefileTokenMarker, ManifestTokenMarker, PatchTokenMarker, PerlTokenMarker, PHPTokenMarker, PropsTokenMarker, PythonTokenMarker, RubyTokenMarker, ShellScriptTokenMarker, SPARQLTokenMarker, SQLTokenMarker, TeXTokenMarker, TextTokenMarker, TurtleTokenMarker, XMLTokenMarker, XPathTokenMarker
```
public abstract class TokenMarker
extends java.lang.Object
```
A token marker that splits lines of text into tokens. Each token carries a length field and an indentification tag that can be mapped to a color for painting that token. For performance reasons, the linked list of tokens is reused after each line is tokenized. Therefore, the return value of markTokens should only be used for immediate painting. Notably, it cannot be cached.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

class TokenMarker.LineInfo
Inner class for storing information about tokenized lines.

Nested Classes
Modifier and Type	Class and Description
`class`	`TokenMarker.LineInfo` Inner class for storing information about tokenized lines.

Field Summary

Fields
Modifier and Type	Field and Description
`protected int`	`currentTokenOffset` The current offset.
`protected Token`	`firstToken` The first token in the list.
`protected int`	`lastLine` The last tokenized line.
`protected Token`	`lastToken` The last token in the list.
`protected int`	`length` The number of lines in the model being tokenized.
`protected int`	`lineIndex` The current line index.
`protected TokenMarker.LineInfo[]`	`lineInfo` An array for storing information about lines.
`protected boolean`	`nextLineRequested` True if the next line should be painted.
`protected int`	`offsetGap` The offset gap for the current line.
`protected SyntaxListener`	`syntaxListener` The syntax lixtener.

Constructor Summary

Constructors
Modifier Constructor and Description

protected TokenMarker()
Creates a new TokenMarker.

Constructors
Modifier	Constructor and Description
`protected`	`TokenMarker()` Creates a new `TokenMarker`.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`addToken(int length, byte id, boolean externalTokenizer)` Adds a token to the token list.
`void`	`deleteLines(int index, int lines)` Informs the token marker that line have been deleted from the document.
`protected void`	`ensureCapacity(int index)` Ensures that the `lineInfo` array can contain the specified index.
`int`	`getLineCount()` Returns the number of lines in this token marker.
`int`	`getLineIndex()` Return the current line index.
`int`	`getMaxLineWidth(int start, int len)` Returns the maximum line width in the specified line range.
`int`	`getOffsetGap()` Return the offset gap for the current line.
`SyntaxListener`	`getSyntaxListener()` Return the syntax listener.
`void`	`insertLines(int index, int lines)` Informs the token marker that lines have been inserted into the document.
`boolean`	`isNextLineRequested()` Returns true if the next line should be repainted.
`Token`	`markTokens(javax.swing.text.Segment line, int lineIndex)` A wrapper for the lower-level `markTokensImpl` method that is called to split a line up into tokens.
`Token`	`markTokens(javax.swing.text.Segment line, int lineIndex, boolean externalTokenizer)` A wrapper for the lower-level `markTokensImpl` method that is called to split a line up into tokens.
`protected abstract byte`	`markTokensImpl(byte token, javax.swing.text.Segment line, int lineIndex, boolean externalTokenizer)` An abstract method that splits a line up into tokens.It should parse the line, and call `addToken()` to add syntax tokens to the token list.
`boolean`	`setLineWidth(int lineIndex, int width)` Store the width of a line, in pixels.
`void`	`setOffsetGap(int offsetGap)` Set the offset gap for the current line.
`void`	`setSyntaxListener(SyntaxListener syntaxListener)` Set the syntax listener.
`boolean`	`supportsMultilineTokens()` Returns true if the token marker supports tokens that span multiple lines.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - firstToken
```
protected Token firstToken
```
    The first token in the list. This should be used as the return value from markTokens().
  - lastToken
```
protected Token lastToken
```
    The last token in the list. New tokens are added here. This should be set to null before a new line is to be tokenized.
  - lineInfo
```
protected TokenMarker.LineInfo[] lineInfo
```
    An array for storing information about lines. It is enlarged and shrunk automatically by the insertLines() and deleteLines() methods.
  - length
```
protected int length
```
    The number of lines in the model being tokenized. This can be less than the length of the lineInfo array.
  - lastLine
```
protected int lastLine
```
    The last tokenized line.
  - nextLineRequested
```
protected boolean nextLineRequested
```
    True if the next line should be painted.
  - syntaxListener
```
protected SyntaxListener syntaxListener
```
    The syntax lixtener.
  - lineIndex
```
protected int lineIndex
```
    The current line index.
  - currentTokenOffset
```
protected int currentTokenOffset
```
    The current offset.
  - offsetGap
```
protected int offsetGap
```
    The offset gap for the current line.
- Constructor Detail
  - TokenMarker
```
protected TokenMarker()
```
    Creates a new TokenMarker. This DOES NOT create a lineInfo array; an initial call to insertLines(int, int) does that.
- Method Detail
  - setSyntaxListener
```
public void setSyntaxListener(SyntaxListener syntaxListener)
```
    Set the syntax listener.
    
    Parameters:
    
    syntaxListener - the syntax listener
  - getSyntaxListener
```
public SyntaxListener getSyntaxListener()
```
    Return the syntax listener.
    
    Returns:
    
    the syntax listener
  - setOffsetGap
```
public void setOffsetGap(int offsetGap)
```
    Set the offset gap for the current line.
    
    Parameters:
    
    offsetGap - the offset gap
  - getOffsetGap
```
public int getOffsetGap()
```
    Return the offset gap for the current line.
    
    Returns:
    
    the offset gap
  - getLineIndex
```
public int getLineIndex()
```
    Return the current line index.
    
    Returns:
    
    the line index
  - markTokens
```
public Token markTokens(javax.swing.text.Segment line,
                        int lineIndex)
```
    A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.
    
    Parameters:
    
    line - The line
    
    lineIndex - The line number
    
    Returns:
    
    the Token
  - markTokens
```
public Token markTokens(javax.swing.text.Segment line,
                        int lineIndex,
                        boolean externalTokenizer)
```
    A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.
    
    Parameters:
    
    line - The line
    
    lineIndex - The line number
    
    externalTokenizer - true when using an external tokenizer
    
    Returns:
    
    the Token
  - markTokensImpl
```
protected abstract byte markTokensImpl(byte token,
                                       javax.swing.text.Segment line,
                                       int lineIndex,
                                       boolean externalTokenizer)
```
    An abstract method that splits a line up into tokens.It should parse the line, and call addToken() to add syntax tokens to the token list. Then, it should return the initial token type for the next line. For example if the current line contains the start of a multiline comment that doesn't end on that line, this method should return the comment token type so that it continues on the next line.
    
    Parameters:
    
    token - The initial token type for this line
    
    line - The line to be tokenized
    
    lineIndex - The index of the line in the document, starting at 0
    
    externalTokenizer - true for an external tokenizer
    
    Returns:
    
    The initial token type for the next line
  - supportsMultilineTokens
```
public boolean supportsMultilineTokens()
```
    Returns true if the token marker supports tokens that span multiple lines. If this is true, the object using this token marker is required to pass all lines in the document to the markTokens() method (in turn). The default implementation returns true. It should be overridden to return false on simpler token markers for increased speed.
    
    Returns:
    
    true
  - insertLines
```
public void insertLines(int index,
                        int lines)
```
    Informs the token marker that lines have been inserted into the document. This inserts a gap in the lineInfo array.
    
    Parameters:
    
    index - The first line number
    
    lines - The number of lines
  - deleteLines
```
public void deleteLines(int index,
                        int lines)
```
    Informs the token marker that line have been deleted from the document. This removes the lines in question from the lineInfo array.
    
    Parameters:
    
    index - The first line number
    
    lines - The number of lines
  - getLineCount
```
public int getLineCount()
```
    Returns the number of lines in this token marker.
    
    Returns:
    
    the number of lines in this token marker
  - isNextLineRequested
```
public boolean isNextLineRequested()
```
    Returns true if the next line should be repainted. This will return true after a line has been tokenized that starts a multiline token that continues onto the next line.
    
    Returns:
    
    true if the next line should be repainted
  - ensureCapacity
```
protected void ensureCapacity(int index)
```
    Ensures that the lineInfo array can contain the specified index. This enlarges it if necessary. No action is taken if the array is large enough already.
    It should be unnecessary to call this under normal circumstances; insertLine() should take care of enlarging the line info array automatically.
    
    Parameters:
    
    index - The array index
  - addToken
```
protected void addToken(int length,
                        byte id,
                        boolean externalTokenizer)
```
    Adds a token to the token list.
    
    Parameters:
    
    length - The length of the token
    
    id - The id of the token
    
    externalTokenizer - true for an external tokenizer
  - setLineWidth
```
public boolean setLineWidth(int lineIndex,
                            int width)
```
    Store the width of a line, in pixels.
    
    Parameters:
    
    lineIndex - The line number
    
    width - The width
    
    Returns:
    
    the width of the line
  - getMaxLineWidth
```
public int getMaxLineWidth(int start,
                           int len)
```
    Returns the maximum line width in the specified line range.
    
    Parameters:
    
    start - The first line
    
    len - The number of lines
    
    Returns:
    
    the maximum line width

Class TokenMarker

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

firstToken

lastToken

lineInfo

length

lastLine

nextLineRequested

syntaxListener

lineIndex

currentTokenOffset

offsetGap

Constructor Detail

TokenMarker

Method Detail

setSyntaxListener

getSyntaxListener

setOffsetGap

getOffsetGap

getLineIndex

markTokens

markTokens

markTokensImpl

supportsMultilineTokens

insertLines

deleteLines

getLineCount

isNextLineRequested

ensureCapacity

addToken

setLineWidth

getMaxLineWidth