Commodore PET BASIC Tokenizer: A Curious Bug

This article explores a quirky bug in early Commodore PET BASIC tokenizers stemming from their whitespace handling. Early BASIC interpreters ignored spaces between keywords, leading to 'LET THEN' being interpreted as 'LETHEN', resulting in syntax errors. The article delves into the BASIC tokenization process, explaining why ignoring whitespace improved efficiency, and dissects the Commodore BASIC 1.0 tokenizer code. It ultimately reveals the root cause of the bug and its fix in later versions.