AI Hype Gets Wrecked by Real-World Job Test

An analysis of the research paper "Remote Labor Index: Measuring AI Automation of Remote Work" is presented, focusing on how frontier AI systems performed when evaluated against actual paid remote freelance projects. The study utilized a benchmark comprising 240 real freelance projects across 23 categories, using human-created deliverables as the standard for measurement, rather than narrow, abstract tests. The research aimed to assess the practical automation capabilities of current artificial intelligence agents in real-world work scenarios.

The findings indicated that the performance of current AI agents was low, with the highest-scoring system achieving only a 2.5% automation rate across the tested projects. Researchers documented various failure modes, such as inconsistent quality, incompleteness, and corrupted outputs. The benchmark represented work valued at over USD 140,000, intended to reflect the complexity of actual commissioned work and challenge exaggerated claims within the current artificial intelligence hype cycle regarding its immediate impact on labor automation.