Researchers studied what happens when using AI Code and found way more errors in the code
Ever notice how more and more major outages are being caused in large software companies, while others seem to be going strong? As it turns out, our AI habits might have a lot to do with those issues.
Research published in the Journal of Systems and Software found evidence that it does not matter your development experience, everyone is more likely to be highly likely to use AI to help generate code. But it's not all good news.
CodeRabbit AI found that AI code creates 1.7x more problems. That's not just a small gap. The team of mostly Pro-AI researchers found that people with AI code usage habits face a mind-blowing 170 percent higher risk of producing critical issues when performing pull requests (PR).
But those who don't use AI to generate code, they found, showed both fewer critical errors and fewer major issues when submitting code to GitHub repos.
To conduct the test, researchers at CodeRabbit AI reviewed 470 open-source GitHub pull requests. That includes 320 AI-co-authored PRs and 150 human-only PRs. The disturbing results found that error & exception handling mistakes were nearly two times more common.
The study revealed that those with more AI-generated code tended to also have more security issues. When they did security-related work, the AI code tended to have 2.74 times as many vulnerabilities. As a result, they failed to handle passwords safetly 188% more frequently and produced 2.74 times more XSS vulnerabilities, while increasing anxiety woes for security consultants like me.
Going forward, researchers will need to replicate the study with a much larger sample size to test the results. Still, the first-of-its-kind study makes one thing clear: next time you’re stressed, it wouldn’t be a bad idea to delete your saved passwords and stop trusting the software that is running your life.
