TokenMarker

The children classes of the abstract TokenMarker class are responsible to create the Tokens (the syntax highlighting for a text.

Association of a TokenMarker to a file type

Associating a TokenMarker to a file type is performed through the SyntaxMapper class with the xml ffile descripting the association. For example for a C++ syntax:

   <file name="c++" path="org.jeditor.scripts.tokenmarkers.CCTokenMarker" type="code">
      <altName name="cpp" />
      <extension extension="cpp" />
      <extension extension="cc" />
      <extension extension="hpp" />
   </file>

TokenMarker implementation

The TokenMarker.markTokensImpl(byte, Segment, int, boolean) method is the only abstract method you have to implement.

For example, the BasicTokenMarker creates only one Token.NULL token (default Token with no syntax) for the whole text:

   public class BasicTokenMarker extends TokenMarker {
      public BasicTokenMarker() {
      }

      /**
      * Splits a line up into tokens.
      *
      * @param token The initial token type for this line
      * @param line The line to be tokenized
      * @param lineIndex The index of the line in the document, starting at 0
      * @return The initial token type for the next line
      */
      @Override
      protected byte markTokensImpl(byte token, Segment line, int lineIndex, boolean externalTokenizer) {
        char[] array = line.array;
        int offset = line.offset;
        int length = line.count + offset;
        this.currentTokenOffset = offset;

        int textLength = 0;

        for (int i = offset; i < length; i++) {
          char c = array[i];
          textLength++;
        }

        if (textLength != 0) {
          addToken(textLength, Token.NULL, externalTokenizer);
        }
        return token;
      }
   }