New Benchmark Exposes the Automation Bottleneck in OCR: Achieving 98% Precision

The influx of new OCR players like Mistral and Andrew Ng's offerings makes it hard for enterprises to distinguish genuine advancements from hype. Existing benchmarks focus on OCR accuracy and information extraction, neglecting automation levels. Nanonets introduces a new benchmark emphasizing automation at 98% precision. Using a dataset of 1000 images and 16,639 annotated data points, they measure model performance based on confidence scores – the proportion of data accurately processed without human intervention. While LLMs excel in overall accuracy, reliable confidence scores remain elusive. Gemini 2.0 Flash achieved 98% precision but automated only 8% of the data. This benchmark aims to help enterprises find solutions that truly reduce manual effort in document processing.