Do LLMs Understand Nulls? Probing the Internal Representations of Code-Generating Models

Large language models (LLMs) have shown remarkable progress in code generation, but their true understanding of code remains a question. This work investigates LLMs' comprehension of nullability in code, employing both external evaluation (code completion) and internal probing (model activation analysis). Results reveal LLMs learn and apply rules about null values, with performance varying based on rule complexity and model size. The study also illuminates how LLMs internally represent nullability and how this understanding evolves during training.