Vibe Coding a Lox Compiler, Part 2: Tokenization
Written with AI This post — and all the code behind it — was built with Claude Code. Read Part 0 for the full story of how this AI-assisted project came together. In Part 1 we introduced the Lox language and took a high-level tour of the full compiler pipeline — from raw source text all the way to native code. Now it’s time to go deep on the first phase of that pipeline: tokenization. This is where a plain string of characters becomes something a parser can actually reason about. ...