Self-Proclaimed 'First AI Software Engineer' Fails Miserably in Real-World Tests
2025-01-26
Devin, marketed as the first AI software engineer, has fallen short of expectations in recent evaluations. Despite claims of building and deploying apps end-to-end and autonomously fixing bugs, Devin succeeded in only 3 out of 20 tasks. Testers found Devin struggled with straightforward tasks, getting stuck in technical dead-ends and pursuing impossible solutions. While offering a polished user experience, its infrequent success and tendency to waste time on unachievable goals highlight the limitations of current AI technology and raise concerns about the hype surrounding AI tools.