Security risks in AI-developed applications

AI has collapsed the time it takes to build software. It has not collapsed the time it takes to make software safe. Those are two different jobs, and only one of them got faster.

An application assembled in an afternoon can look finished, demo cleanly, and still be missing everything that keeps it standing once real users and real data arrive. The build is visible and quick. The security work is invisible, and was never on screen, so it is the part that quietly gets skipped.

Why the result is insecure by default

A model generates the code most likely to satisfy the request in front of it. “Make the login work” produces a login that works. It does not produce a login that resists someone actively trying to break it, because resisting attackers was not what the prompt asked for and is not what the demo rewards.

That leaves a predictable gap, and the person prompting often cannot see it either. The screen shows a working feature, not the missing rate limit, the over-broad database rule, or the secret sitting in the bundle. Research on AI coding assistants has repeatedly found the same pattern: people ship less secure code with an assistant while feeling more confident that it is secure. Speed and false confidence are a bad combination to hand an attacker.

The risk lives around the code, not in it

Most of the danger in an AI-built application is not in the lines the model wrote. It is in the layer around them, the decisions that never show up in a demo:

• Secrets in the open. API keys and database credentials committed to the repository or shipped in client-side code, where anyone can read them.
• Access left wide open. Authentication that can be bypassed, or database rules that never actually restricted anything, so one account can reach everyone’s data.
• Untrusted input trusted anyway. User input, and increasingly model output, used to build a query or a command without being treated as hostile.
• Dependencies nobody vetted. Packages pulled in on the model’s suggestion, carrying their own vulnerabilities and their own supply chain.
• No limits and no one watching. Nothing to stop one bad actor running up the bill or taking the service down, and alerts firing into an inbox no one reads.
• Backups nobody has restored. A backup you have never tested is a hope, not a backup.

None of these are exotic. They are the boring, checkable things that get missed when nobody was handed the job of checking. Frameworks like the OWASP Top 10 exist precisely because the same short list of failures keeps recurring, whoever, or whatever, wrote the code.

“It works” and “it is safe” are different tests

Code that returns the right answer on the happy path tells you almost nothing about how it behaves on a hostile one. The happy path is what a model optimises for and what a demo shows. The hostile path, the malformed input, the request from an account that should not have access, the dependency with a known flaw, is exactly the path no one exercised.

If no one was ever given responsibility for the unglamorous parts, assume they were skipped.

The check is the point

The answer is not to build with AI less. It is to stop treating “it works” as though it also meant “it is safe.” That has to be someone’s explicit job, and it cannot be the person, or the agent, that built the thing.

Before an AI-built application carries anything you would mind losing, someone who did not write it should be able to answer, in writing:

1. Where do secrets live, and has anyone confirmed none are in the code or the repository?
2. Who can reach the data directly, and what stops them reaching more than they should?
3. Does authentication actually block the people it is meant to block, verified by testing it rather than reading it?
4. Is untrusted input, from users and from models, treated as hostile before it reaches a query or a command?
5. Has a backup been restored end to end, not just taken?
6. If an AI agent has write access to anything live, what is it prevented from doing without a human confirming first?

That is what a gate is for. Ours checks against OWASP, and against ISO 27001, SOC 2 or HIPAA where they apply, before anything earns a production-ready stamp. Fixing these gaps is almost always cheaper than the rebuild people reach for first, because they are configuration and process failures, not proof that the application was worthless. The expensive version is finding out in public.

---

Want your own application checked before it goes to production? Talk to us.