LLMs Fail a Real-World Fact-Check: A Stark Divide in Capabilities

The author tested several large language models (LLMs) on a complex real-world fact-checking task concerning the long-term effects of ADHD medication. Results revealed a significant performance gap: some LLMs accurately cited and summarized real-world documents, while others suffered from severe 'link hallucinations' and source misinterpretations. The author argues that current LLM testing methods are too simplistic and fail to adequately assess their ability to handle complex information, calling for greater attention to this critical issue.