The security of generative AI (genAI) models is becoming a pressing concern as developers increasingly rely on AI-generated code. While genAI has the potential to revolutionize software development, its security shortcomings may erode the trust the technology needs for widespread production use.

Historically, new technologies often overlook security in their early stages, focusing instead on performance or convenience. However, genAI's security vulnerabilities may have far-reaching consequences, particularly as the technology is being used to build critical software infrastructure.

The open-source community has learned this lesson the hard way. For years, open-source developers were complacent about security, relying on the "many eyeballs" theory that assumed that with enough developers reviewing code, all bugs would be shallow. However, the Heartbleed bug in 2014 shattered this myth, and since then, there has been a steady stream of supply chain attacks against Linux and other prominent open-source software.

According to a recent report, open-source malware has increased by 200% since 2023 and is expected to continue rising as developers embed open-source packages into their projects. The report's authors note that open-source malware thrives in ecosystems with low entry barriers, no author verification, high usage, and diverse users.

Adding to the problem is the growing trend of using AI to author bug reports, which can overload project maintainers with low-quality, spammy, or hallucinated security reports. Moreover, genAI platforms like GitHub Copilot can learn from code posted online and pick up bad habits, including bugs and security vulnerabilities.

As genAI is increasingly used to build software, the stakes are much higher. If genAI models are not secure, they can introduce bugs and vulnerabilities into the software they generate, compromising the security of the entire system.

Unfortunately, the companies building large language models, including OpenAI and Meta, are not prioritizing security. According to the newly released AI Safety Index, the industry as a whole is failing to address safety and risk concerns. The best-performing company, Anthropic, earned a C grade, and experts warn that the current approach to AI development is not providing any quantitative guarantees of safety.

However, there is cause for hope. As enterprises begin to demand higher levels of security from genAI vendors, the industry is likely to respond. Already, concerns over genAI security are hampering adoption, and it is up to enterprises to push for better security standards.

In conclusion, the security of genAI models is a pressing concern that needs to be addressed urgently. As the technology continues to gain traction in software development, it is crucial that developers, vendors, and enterprises prioritize security to ensure the trust and widespread adoption of genAI.