A lexer is the first compiler phase to see the source, but it does not understand a complete statement. It reads characters, groups them into lexemes, and emits tokens that the parser can consume. On if (count1 >= 10) total = total + rate2 * 3.5;, maximal munch produces 14 tokens and reuses one symbol-table pointer for both occurrences of total.
What lexical analysis does, and where it stops
In the compiler front end, the scanner sits between the source-character stream and the parser. Lexical Analysis in Compiler Design: Tokens to DFA Scanner explains the broader path from compiler phases to DFA scanning. In if (count1 >= 10) total = total + rate2 * 3.5;, longest match, emitted attributes, and symbol-table reuse are visible character by character.
Its core jobs are to:
recognise character sequences that match token patterns;
skip permitted whitespace and comments;
attach useful attributes to emitted tokens;
record source positions for diagnostics;
report a character that matches no valid pattern.
Consider total = rate2 * 3.5;. The scanner can emit ID, ASSIGN, ID, MUL, REAL_LITERAL, and SEMICOLON. It does not decide whether rate2 was declared or whether its type can be multiplied by a real literal. The parser checks whether the token sequence fits the grammar, while semantic analysis later checks declarations and types.
Token, lexeme, pattern, and attribute
These terms describe different parts of one recognition step. In count1 >= 10, a lexeme is the matched text, a token is its category, a pattern describes its valid shape, and an attribute carries the value or reference needed later.
Lexeme | Token | Pattern | Attribute |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
| value |
|
| identifier-shaped text followed by keyword-table lookup | none |
Many lexemes can share the token class ID. The attribute preserves which identifier was found. Similarly, RELOP carries GE and INT_LITERAL carries 10.
Keyword lookup happens after the identifier-shaped text has been recognised. Therefore, if becomes IF, but ifx becomes one ID. It is not split into IF and ID(x), because the identifier pattern consumes the longer valid lexeme first.
Maximal munch on count1>=10: a three-token DFA trace
A deliberately small toy-language rule set is enough to trace this decision. It is not the complete lexical grammar of C, Java, or any other production language.
Identifiers:
[A-Za-z_][A-Za-z0-9_]*Integers:
[0-9]+Reals:
[0-9]+\.[0-9]+Relational operators:
>=|<=|==|!=|>|<Punctuation:
(,),;Arithmetic operators:
+,*Assignment:
=Whitespace:
[ \t\n]+, with actionskip
Regular expressions state the patterns, and finite automata recognise them as characters arrive. Maximal munch means that, from the current position, the scanner chooses the longest prefix that forms a valid token.
Scan count1>=10. The identifier state consumes all six characters of count1. At >, the scanner is already in an accepting state, but lookahead finds = and reaches the longer accepting lexeme >=. The integer state then consumes both digits of 10.
The exact output is <ID,p1> <RELOP,GE> <INT_LITERAL,10>. It is not count followed by 1, and it is not > followed by =.
Fully worked scan from a source line to 14 tokens
Now scan the complete line:
if (count1 >= 10) total = total + rate2 * 3.5;
Move left to right, take the longest valid match, and perform keyword lookup for identifier-shaped text.
Number | Lexeme | Emitted token and attribute |
|---|---|---|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
5 |
|
|
6 |
|
|
7 |
|
|
8 |
|
|
9 |
|
|
10 |
|
|
11 |
|
|
12 |
|
|
13 |
|
|
14 |
|
|
Spaces match the whitespace rule and are consumed, but they emit no token. The exact stream sent onward is:
<IF> <LPAREN> <ID,p1> <RELOP,GE> <INT_LITERAL,10> <RPAREN> <ID,p2> <ASSIGN> <ID,p2> <PLUS> <ID,p3> <MUL> <REAL_LITERAL,3.5> <SEMICOLON>
In one standard teaching representation, the associated symbol-table snapshot is p1 -> count1, p2 -> total, and p3 -> rate2. Both occurrences of total carry the same pointer, p2.
Symbol tables, skipped text, and lexical errors
On the first sightings of count1, total, and rate2, this scanner associates them with p1, p2, and p3. When total appears again, it reuses p2. This is a useful model, but compiler implementations need not store literals or keywords in exactly the same table.
Skipped input is still valid input under a rule. Spaces and \n match the whitespace pattern and are intentionally discarded. If the language defines a comment rule, the scanner can consume a complete comment while still updating line positions.
Erroneous input is different. In rate2 @ 3.5, the @ is at one-based character position 7. It matches none of the listed toy rules, so the scanner reports a lexical error instead of silently dropping it.
The treatment of 25abc depends on the language specification. One language may reject it as a malformed literal, while another scanner may produce 25 and abc as adjacent tokens. There is no universal split independent of the language rules.
Lexical analysis question patterns and token-count traps
GATE-style lexical-analysis questions commonly ask you to count tokens, distinguish token from lexeme and pattern, select a regular expression or automaton, apply longest match, and separate scanner duties from parser or semantic-analysis duties.
Consider this compact trap:
sum += arr[i++];
Under ordinary C-style token classes, it has exactly 8 tokens:
sum+=arr[i++];
Maximal munch makes += one assignment-operator token and ++ one increment-operator token. Neither is split into single-character operators.
Three checks prevent most mistakes:
apply longest match before splitting;
count every repeated lexeme occurrence, even when two occurrences share one symbol-table entry;
exclude skipped whitespace and comments from the emitted-token count.
Use the solved lexical-analysis MCQs to practise these distinctions. For current GATE scope, confirm the official GATE CS syllabus rather than relying on an old weightage or marks claim.
The short version and the next practice step
A lexeme is the exact matched text.
A pattern is the recognition rule.
A token is the category sent to the parser.
An attribute carries the specific value or reference.
Try deriving all 14 tokens and the p1, p2, p3 symbol table from the worked line without looking back. If you can reproduce both, the core scanning method is in place.
For an ordered Compiler Design route, continue with GATE Guidance by Sanchit Sir. For broader CS foundations, use Zero to Hero. The CS Fundamentals category is the neutral path for browsing related topics.
