Lexical analysis is the process of splitting each line of input into one or more possible sequences of terminal symbols. A terminal symbol is either a syntax terminal from the grammar or from the lexicon, or a regular expression terminal from the grammar. The result of the lexical analysis is a word lattice.
If no lexical analysis of an input sentence exists, the unknown parts will be ignored and the identified input will be parsed. The skipped unknown part, which is always a token is indicated.
The next section define some important terms for lexical analysis. After that some examples of terminal symbols and their analysis will be given.