Lexicon Entry
Lexer / Tokenizer
The first phase of a compiler frontend that reads raw source text and produces a stream of tokens — classified chunks like identifiers, keywords, operators, and literals.
Related Knowledge & Cross-References
Guide
Write a Compiler from Scratch in C: Build a Working Toy Compiler
Learn how to write a compiler from scratch in C. Build a complete toy compiler with a hand-written lexer, recursive descent parser, and code generator targeting x86-64 and ARM64 assembly.
Guide
Compiler Lexer & Parser Demystified: How Compilers Read Your C Code
Learn how a compiler lexer and parser transform C source into tokens and ASTs. Hands-on with Clang's -dump-tokens and -ast-dump across real examples.