Summary: Stanford’s controlled user study (CCS ‘23). Participants with an AI assistant wrote less secure code — and were more likely to believe their code was secure. The vendor-independent empirical anchor the vault was missing.
Sources: raw/Supply-chain-attacks/Aisecurecode.md — Perry, Srivastava, Kumar & Boneh, Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, DOI 10.1145/3576915.3623157
Last updated: 2026-07-29
Why this source matters disproportionately
Every other claim in the vault about AI-generated code quality comes from vendor content — Cycode’s uncited “2.7× vulnerability density,” Orca’s product-adjacent guide, Checkmarx’s CTA-laden checklist (AI-generated code risk).
This is peer-reviewed, controlled, and measures the thing that actually matters: not whether models emit insecure code in a lab, but whether developers using them ship insecure code.
“While recent work has demonstrated that such tools may erroneously produce security mistakes, no study has extensively measured the security risks of AI assistants in the context of how developers choose to use them.”
Method
47 participants, five security-related programming tasks, across three languages (Python, JavaScript, C). Three research questions:
- RQ1 — Do users write more insecure code when given an AI assistant?
- RQ2 — Do users trust AI assistants to write secure code?
- RQ3 — How do users’ language and behaviour when interacting affect vulnerability rates?
Results were modelled “while controlling for a variety of factors like their exposure to security concepts, their previous programming experience, and their student status.”
The two headline findings
1. Less secure code. “Participants with access to an AI assistant wrote insecure solutions more often than those without access to an AI assistant for four of our five programming tasks… users with access to an AI assistant typically produced less secure code.”
2. More confidence. “To make matters worse, participants that were provided access to an AI assistant were more likely to believe that they wrote secure code than those without access.”
The second finding is the one to carry. It is not that the tool is imperfect — it is that the tool inverts the calibration between confidence and correctness. A developer who knows they are unsure will check. A developer who wrongly believes they are secure will not.
“AI code assistants may significantly lower the barrier of entry for non-programmers and increase developer productivity, [but] they may provide inexperienced users a false sense of security.”
This is the mechanism behind OWASP’s LLM09 Misinformation entry, where overreliance “occurs when users place excessive trust in LLM-generated content, failing to verify its accuracy” — measured here rather than asserted.
What made outcomes better
RQ3 is the constructive part, and it maps directly onto prompt discipline:
- Participants “who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”
- “The ability to clearly express your prompts and appropriately rephrase them to get a desired answer was crucial for writing correct and secure code.”
This is the empirical backing for CSA’s assertion that “prompts are now the code design specification” (securing the AI SDLC), and for the whole premise of OpenSSF’s instruction guidance — which cites this paper directly.
What made outcomes worse
“Using previous outputs of the AI Assistant as new prompts can result in security problems being magnified or replicated.”
A feedback loop where the model’s own output becomes its next input, compounding flaws. Structurally identical to two things already in the vault:
- PITAX T32 Echo Chamber, where poisoned framing “self-reinforces” as the model echoes its own context
- AI Act Art 15(4), which requires continuously-learning systems to address “the risk of possibly biased outputs influencing input for future operations (feedback loops)”
The Act mandates managing this for the system; the study shows it happening in the developer’s workflow. Same failure mode, different loop.
Also noted: participants who used the assistant to write secure code “increased the temperature parameter more and gave prompts with more context” — a behavioural signal, not a recommendation.
Caveats worth stating
- 47 participants is a normal size for usable-security research but small in absolute terms.
- CCS ‘23, so the models studied are 2022–23 vintage (Codex, InCoder era). Whether the effect persists with current models is untested here — though the overconfidence finding is about human calibration, which is less likely to have shifted than model capability.
- The authors released their study apparatus for replication, which is unusual and good.
Why it sits in this folder
Physically filed under raw/Supply-chain-attacks/, though topically it belongs with Secure Coding Patterns. Cross-linked both ways; see the log for the folder-scope note.