AI-Generated Website: An Experiment in Skill vs. Knowledge
2024-12-31
Security researcher Nicholas Carlini conducted a twelve-day experiment: rewriting his website homepage and bio daily using a different language model. He found that while models excelled at generating visually stunning webpages, they faltered significantly in factual accuracy. For example, the o1-mini model generated a webpage with 43 statements; 32 were completely false, 9 had major errors, and only 2 were factually correct. This highlights the vast discrepancy between "skill" (generating webpages) and "knowledge" (factual accuracy) in LLMs, underscoring the need for caution when relying on AI-generated content.