2.1 Overview

• Read a stream of characters and group them into tokens

  Examples: keywords, identifiers, constants, operators

• Return the syntactic and semantics information of the tokens
  1. The syntactic information might be an integer code representing the type of the token
  2. The semantics information might be provided indirectly through a pointer to a symbol table holding the information
  3. Use finite state automata for modeling