## mars.assembler
Class Tokenizer

    java.lang.Object
      mars.assembler.Tokenizer

    public class Tokenizer

extends Object

A tokenizer is capable of tokenizing a complete MIPS program, or a given line from a MIPS program. Since MIPS is line-oriented, each line defines a complete statement. Tokenizing is the process of analyzing the input MIPS program for the purpose of recognizing each MIPS language element. The types of language elements are known as "tokens". MIPS tokens are defined in the TokenTypes class.
Example:
The MIPS statement `here: lw $t3, 8($t4) #load third member of array`
generates the following token list
IDENTIFIER, COLON, OPERATOR, REGISTER_NAME, COMMA, INTEGER_5, LEFT_PAREN, REGISTER_NAME, RIGHT_PAREN, COMMENT

Author:
  Pete Sanderson

Constructor Summary
`Tokenizer()` Simple constructor.
`Tokenizer(MIPSprogram program)` Constructor for use with existing MIPSprogram.

Method Summary
`ErrorList` `getErrors()` Fetch this Tokenizer's error list.
`ArrayList` `tokenize(MIPSprogram p)` Will tokenize a complete MIPS program.
`TokenList` `tokenizeExampleInstruction(String example)` Used only to create a token list for the example provided with each instruction specification.
`TokenList` `tokenizeLine(int lineNum, String theLine)` Will tokenize one line of source code.
`TokenList` `tokenizeLine(int lineNum, String theLine, ErrorList callerErrorList)` Will tokenize one line of source code.
`TokenList` `tokenizeLine(int lineNum, String theLine, ErrorList callerErrorList, boolean doEqvSubstitutes)` Will tokenize one line of source code.
`TokenList` `tokenizeLine(MIPSprogram program, int lineNum, String theLine, boolean doEqvSubstitutes)` Will tokenize one line of source code.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### Tokenizer

    public Tokenizer()

  Simple constructor. Initializes empty error list.

### Tokenizer

    public Tokenizer(MIPSprogram program)

  Constructor for use with existing MIPSprogram. Designed to be used with Macro feature.

  Parameters:
    `program` - A previously-existing MIPSprogram object or null if none.

Method Detail

### tokenize

    public ArrayList tokenize(MIPSprogram p)
                       throws ProcessingException

  Will tokenize a complete MIPS program. MIPS is line oriented (not free format), so we will be line-oriented too.

  Parameters:
    `p` - The MIPSprogram to be tokenized.
  Returns:
    An ArrayList representing the tokenized program. Each list member is a TokenList that represents a tokenized source statement from the MIPS program.
  Throws:
    `ProcessingException`

### tokenizeExampleInstruction

    public TokenList tokenizeExampleInstruction(String example)
                                         throws ProcessingException

  Used only to create a token list for the example provided with each instruction specification.

  Parameters:
    `example` - The example MIPS instruction to be tokenized.
  Returns:
    An TokenList representing the tokenized instruction. Each list member is a Token that represents one language element.
  Throws:
    `ProcessingException` - This occurs only if the instruction specification itself contains one or more lexical (i.e. token) errors.

### tokenizeLine

    public TokenList tokenizeLine(int lineNum,
                                  String theLine)

  Will tokenize one line of source code. If lexical errors are discovered, they are noted in an ErrorMessage object which is added to the ErrorList. Will NOT throw an exception yet because we want to persevere beyond first error.

  Parameters:
    `lineNum` - line number from source code (used in error message)
    `theLine` - String containing source code
  Returns:
    the generated token list for that line

### tokenizeLine

    public TokenList tokenizeLine(int lineNum,
                                  String theLine,
                                  ErrorList callerErrorList)

  Will tokenize one line of source code. If lexical errors are discovered, they are noted in an ErrorMessage object which is added to the provided ErrorList instead of the Tokenizer's error list. Will NOT throw an exception.

  Parameters:
    `lineNum` - line number from source code (used in error message)
    `theLine` - String containing source code
    `callerErrorList` - errors will go into this list instead of tokenizer's list.
  Returns:
    the generated token list for that line

### tokenizeLine

    public TokenList tokenizeLine(int lineNum,
                                  String theLine,
                                  ErrorList callerErrorList,
                                  boolean doEqvSubstitutes)

  Will tokenize one line of source code. If lexical errors are discovered, they are noted in an ErrorMessage object which is added to the provided ErrorList instead of the Tokenizer's error list. Will NOT throw an exception.

  Parameters:
    `lineNum` - line number from source code (used in error message)
    `theLine` - String containing source code
    `callerErrorList` - errors will go into this list instead of tokenizer's list.
    `doEqvSubstitutse` - boolean param set true to perform .eqv substitutions, else false
  Returns:
    the generated token list for that line

### tokenizeLine

    public TokenList tokenizeLine(MIPSprogram program,
                                  int lineNum,
                                  String theLine,
                                  boolean doEqvSubstitutes)

  Will tokenize one line of source code. If lexical errors are discovered, they are noted in an ErrorMessage object which is added to the provided ErrorList instead of the Tokenizer's error list. Will NOT throw an exception.

  Parameters:
    `program` - MIPSprogram containing this line of source
    `lineNum` - line number from source code (used in error message)
    `theLine` - String containing source code
    `doEqvSubstitutes` - boolean param set true to perform .eqv substitutions, else false
  Returns:
    the generated token list for that line

### getErrors

    public ErrorList getErrors()

  Fetch this Tokenizer's error list.

  Returns:
    the error list