## mars.venus.editors.jeditsyntax.tokenmarker
Class TokenMarker

    java.lang.Object
      mars.venus.editors.jeditsyntax.tokenmarker.TokenMarker

Direct Known Subclasses:
  MIPSTokenMarker

    public abstract class TokenMarker

extends Object

A token marker that splits lines of text into tokens. Each token carries a length field and an indentification tag that can be mapped to a color for painting that token.

For performance reasons, the linked list of tokens is reused after each line is tokenized. Therefore, the return value of `markTokens` should only be used for immediate painting. Notably, it cannot be cached.

Author:
  Slava Pestov
See Also:
  `org.syntax.jedit.Token`

Nested Class Summary
`class` `TokenMarker.LineInfo` Inner class for storing information about tokenized lines.

Field Summary
`protected Token` `firstToken` The first token in the list.
`protected int` `lastLine` The last tokenized line.
`protected Token` `lastToken` The last token in the list.
`protected int` `length` The number of lines in the model being tokenized.
`protected TokenMarker.LineInfo[]` `lineInfo` An array for storing information about lines.
`protected boolean` `nextLineRequested` True if the next line should be painted.

Constructor Summary
`protected` `TokenMarker()` Creates a new `TokenMarker`.

Method Summary
`protected void` `addToken(int length, byte id)` Adds a token to the token list.
`void` `deleteLines(int index, int lines)` Informs the token marker that line have been deleted from the document.
`protected void` `ensureCapacity(int index)` Ensures that the `lineInfo` array can contain the specified index.
`int` `getLineCount()` Returns the number of lines in this token marker.
`ArrayList` `getTokenExactMatchHelp(Token token, String tokenText)` Construct and return any appropriate help information for the given token.
`ArrayList` `getTokenPrefixMatchHelp(String line, Token tokenList, Token tokenAtOffset, String tokenText)` Construct and return any appropriate help information for the given token or "token prefix".
`void` `insertLines(int index, int lines)` Informs the token marker that lines have been inserted into the document.
`boolean` `isNextLineRequested()` Returns true if the next line should be repainted.
`Token` `markTokens(Segment line, int lineIndex)` A wrapper for the lower-level `markTokensImpl` method that is called to split a line up into tokens.
`protected abstract byte` `markTokensImpl(byte token, Segment line, int lineIndex)` An abstract method that splits a line up into tokens.
`boolean` `supportsMultilineTokens()` Returns if the token marker supports tokens that span multiple lines.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### firstToken

    protected Token firstToken

  The first token in the list. This should be used as the return value from `markTokens()`.

### lastToken

    protected Token lastToken

  The last token in the list. New tokens are added here. This should be set to null before a new line is to be tokenized.

### lineInfo

    protected TokenMarker.LineInfo[] lineInfo

  An array for storing information about lines. It is enlarged and shrunk automatically by the `insertLines()` and `deleteLines()` methods.

### length

    protected int length

  The number of lines in the model being tokenized. This can be less than the length of the `lineInfo` array.

### lastLine

    protected int lastLine

  The last tokenized line.

### nextLineRequested

    protected boolean nextLineRequested

  True if the next line should be painted.

Constructor Detail

### TokenMarker

    protected TokenMarker()

  Creates a new `TokenMarker`. This DOES NOT create a lineInfo array; an initial call to `insertLines()` does that.

Method Detail

### markTokens

    public Token markTokens(Segment line,
                            int lineIndex)

  A wrapper for the lower-level `markTokensImpl` method that is called to split a line up into tokens.

  Parameters:
    `line` - The line
    `lineIndex` - The line number

### markTokensImpl

    protected abstract byte markTokensImpl(byte token,
                                           Segment line,
                                           int lineIndex)

  An abstract method that splits a line up into tokens. It should parse the line, and call `addToken()` to add syntax tokens to the token list. Then, it should return the initial token type for the next line.

  For example if the current line contains the start of a multiline comment that doesn't end on that line, this method should return the comment token type so that it continues on the next line.

  Parameters:
    `token` - The initial token type for this line
    `line` - The line to be tokenized
    `lineIndex` - The index of the line in the document, starting at 0
  Returns:
    The initial token type for the next line

### supportsMultilineTokens

    public boolean supportsMultilineTokens()

  Returns if the token marker supports tokens that span multiple lines. If this is true, the object using this token marker is required to pass all lines in the document to the `markTokens()` method (in turn).

  The default implementation returns true; it should be overridden to return false on simpler token markers for increased speed.

### insertLines

    public void insertLines(int index,
                            int lines)

  Informs the token marker that lines have been inserted into the document. This inserts a gap in the `lineInfo` array.

  Parameters:
    `index` - The first line number
    `lines` - The number of lines

### deleteLines

    public void deleteLines(int index,
                            int lines)

  Informs the token marker that line have been deleted from the document. This removes the lines in question from the `lineInfo` array.

  Parameters:
    `index` - The first line number
    `lines` - The number of lines

### getLineCount

    public int getLineCount()

  Returns the number of lines in this token marker.

### isNextLineRequested

    public boolean isNextLineRequested()

  Returns true if the next line should be repainted. This will return true after a line has been tokenized that starts a multiline token that continues onto the next line.

### getTokenExactMatchHelp

    public ArrayList getTokenExactMatchHelp(Token token,
                                            String tokenText)

  Construct and return any appropriate help information for the given token. This default definition returns null; override it in language-specific subclasses.

  Parameters:
    `token` - the pertinent Token object
    `tokenText` - the source String that matched to the token
  Returns:
    ArrayList containing PopupHelpItem objects, one per match.

### getTokenPrefixMatchHelp

    public ArrayList getTokenPrefixMatchHelp(String line,
                                             Token tokenList,
                                             Token tokenAtOffset,
                                             String tokenText)

  Construct and return any appropriate help information for the given token or "token prefix". Will match instruction prefixes, e.g. "s" matches "sw". This default definition returns null; override it in language-specific subclasses.

  Parameters:
    `line` - String containing current line
    `tokenList` - first Token on the current line
    `token` - the pertinent Token object
    `tokenText` - the source String that matched to the token
  Returns:
    ArrayList containing PopupHelpItem objects, one per match.

### ensureCapacity

    protected void ensureCapacity(int index)

  Ensures that the `lineInfo` array can contain the specified index. This enlarges it if necessary. No action is taken if the array is large enough already.

  It should be unnecessary to call this under normal circumstances; `insertLine()` should take care of enlarging the line info array automatically.

  Parameters:
    `index` - The array index

### addToken

    protected void addToken(int length,
                            byte id)

  Adds a token to the token list.

  Parameters:
    `length` - The length of the token
    `id` - The id of the token