LLMs Fail at Complex OCR: Why Large Language Models Struggle with PDFs

Pulse, a company aiming to extract data from spreadsheets and PDFs, discovered a critical limitation in using Large Language Models (LLMs) for OCR. While LLMs excel at text generation and summarization, they falter significantly when dealing with complex PDFs and tables. The probabilistic nature of LLMs and their abstract image processing lead to hallucinations, data loss, and misinterpretations, posing significant risks, especially with financial and medical data. Furthermore, LLMs are vulnerable to prompt injection attacks, raising security and ethical concerns. Pulse ultimately abandoned LLMs for OCR and is developing a custom solution integrating traditional computer vision algorithms and vision transformers.