A controversy is brewing in the academic AI community after at least three AI labs submitted AI-generated studies to workshops at the International Conference on Learning Representations (ICLR), a prestigious conference focused on AI research.

The labs in question, Sakana, Intology, and Autoscience, claimed to have used AI to generate studies that were accepted to ICLR workshops. However, the approach taken by Intology and Autoscience has drawn criticism from AI academics, who argue that the labs co-opted the scientific peer review process without consent.

Sakana, on the other hand, informed ICLR leaders before submitting its AI-generated papers and obtained the peer reviewers' consent. This distinction is crucial, as peer review is a time-consuming and labor-intensive process that relies heavily on volunteer efforts. According to a recent Nature survey, 40% of academics spend two to four hours reviewing a single study, and the number of papers submitted to AI conferences is increasing rapidly.

The controversy highlights a growing issue in academia, where AI-generated copy is becoming increasingly prevalent. One analysis found that between 6.5% and 16.9% of papers submitted to AI conferences in 2023 likely contained synthetic text. However, the use of peer review to benchmark and advertise AI technology is a relatively new occurrence, and many academics are skeptical about the value of AI-generated papers.

Intology's decision to tout its ICLR results on social media, including claims that workshop reviewers praised one of its AI-generated study's "clever idea[s]," has been met with criticism from academics who argue that submitting AI-generated papers without consent shows a "lack of respect for human reviewers' time." Ashwinee Panda, a postdoctoral fellow at the University of Maryland, noted that Sakana had reached out to him before submitting its papers, but Intology did not.

Sakana itself has acknowledged the limitations of its AI-generated papers, admitting that its AI made "embarrassing" citation errors and that only one out of the three papers submitted would have met the bar for conference acceptance. The company ultimately withdrew its ICLR paper in the interest of transparency and respect for ICLR convention.

The controversy has sparked a broader discussion about the need for regulated, compensated evaluations of AI-generated studies. Alexander Doria, the co-founder of AI startup Pleias, argues that a "regulated company/public agency" should perform "high-quality" AI-generated study evaluations for a price, with researchers fully compensated for their time. This approach would help to ensure that AI-generated papers are held to the same standards as human-authored research, while also respecting the time and effort of peer reviewers.

As the use of AI-generated content continues to grow, the academic community must grapple with the implications of this technology on the peer review process. The controversy at ICLR serves as a warning that the unchecked use of AI-generated studies can undermine the integrity of academic research, and that new approaches to evaluation and compensation are needed to ensure the continued validity of scientific peer review.