You're not first, and here's the lever
Part of my GORED series, where I point small local language models at my own web apps and measure, honestly, whether they can break them. The harness is t0ul/GORED and these write-ups live in t0ul/ai-security-engineering. Every number comes from a kept, hash-verified trace.
Intro
By the end of the zeros I had talked myself into two bad conclusions at once: that I had built the thing wrong, and that the idea was dumb to begin with. Both felt true at two in the morning. Neither is, and the thing that convinced me was sitting in the research the whole time, which I should have read first instead of last.
A whole field has walked this exact path. They hit the same wall, they reinvented the same fixes I reinvented, and, importantly, they found the one lever that actually moves the number I could not move. This post is me reading that literature honestly, admitting where it says I was right, admitting where it says I was slow, and pointing at the lever.
I. The field already mapped this
There is a survey that sorts this whole area into four phases, and reading it felt like reading my own lab notebook back to myself. Phase one is text-only: a model reasons about a pentest while a human copies output back and forth. Phase two is a single agent with tools, and the survey names its exact failure mode, the one I hit, which is that the agent drowns in its own accumulated context. Phase three is multi-agent, a planner and doers with memory kept outside the model. Phase four is the trained one.
What struck me is how much of my own design I found already described in phases two and three, arrived at by other people for the same reasons. The single most important decision I made, keeping the thing that decides “did it work” completely separate from the model, is the Naptime and Big Sleep design: scoped tools plus a non-model controller that checks the success condition, so the model never gets to grade its own exam. That approach found a real, previously unknown bug in SQLite. Another project, XBOW, pairs short-lived focused agents with an independent validator that runs a harmless proof and emits a reproducible script, which is exactly the “a finding without a repro is just a claim” rule I kept insisting on. I did not invent these. I rediscovered them, which is a much less lonely feeling than I had at two in the morning.
II. The lever: stop prompting, start fine-tuning
Here is the part I did not know, and it is the answer to “what would actually move this.”
Every lever I reached for in the earlier posts was a prompting lever: scope the tools, sharpen the brief, add more tries, add more models. The research is blunt about those. They are not where the gains are. The gains come from fine-tuning a small model on the task, and the result that made me put my coffee down is a project called Pentest-R1. They took an 8B model, the size class that was failing for me, and trained it with reinforcement learning on thousands of real pentest walkthroughs. On a shared benchmark it went from three percent to twenty-four percent, roughly an eight-fold jump, and in doing so it beat GPT-4o, a model vastly larger than it. They did this on two GPUs, a single machine.
And it is not a one-off. Other trained small models in this space keep landing in the same place: a fine-tuned 7B matching a 70B, small models beating ones ten times their size. The thesis underneath my whole project, that a small local model could be the weapon, turns out to be correct. I just had the wrong verb. Not prompt the small model. Train it.
III. So why didn’t I just do that?
Because fine-tuning an 8B with reinforcement learning needs real GPUs, the kind Pentest-R1 used two of, and the shared benchmarks that would let me measure honestly are built for Linux on amd64 with a container orchestrator. My entire lab is a 16 GB Apple-Silicon laptop. It is a wonderful machine and it cannot be that host. I confirmed this the hard way, trying to stand up the benchmark locally and watching it grind for sixteen minutes on a single test before I accepted that the architecture was simply wrong for it.
I want to be honest about the shape of that, because it is the real lesson of the whole series and it is not a technical one. The hardware ceiling was knowable on day one. The research that names fine-tuning as the lever, and names the GPU requirement, existed before I started. If I had read it first, I would have known within an afternoon that the interesting version of this project lives on a GPU box and the laptop version tops out exactly where mine topped out. I read it last. The sixty hours between those two points bought me a very thorough, very honest map of a wall, which is worth something, but it is not worth sixty hours.
IV. What is actually worth keeping
Two things came out of this that I would not throw away.
The first is the recon half, which I already gave its own post. Small local models are genuinely good at reading an app: mapping its surface, telling an API from a page, digging endpoints out of a single-page app’s JavaScript. That half works today, on a laptop, and it is useful.
The second is EVA, the cockpit I built to run all of this. It turned a pile of command-line sweeps into something I could actually watch and reason about: every model’s runs scored side by side, each one replayable with its hash chain intact.
The comparison lab. Each row is a model paired with a skill, scored on real target bytes, sortable and pinnable. This is the view that made “which model, under which setup” a question I could answer instead of guess.
Studio, where the stages, the skills, and the tool inventory are authored. The harness stopped being a wall of flags and became something I could tune and re-run.
As for the project itself, the forward path is clear now, and it is all on the other side of the hardware line: run the shared benchmark on a proper Linux box, measure across many trials instead of trusting single runs, and then do the thing the research actually points at, which is to fine-tune a small model rather than keep prompting one. The self-consistency ensemble I designed and never ran belongs on that machine too.
Conclusion
The honest ledger is short. A small local model, prompted, cannot discover a vulnerability in your app, and no amount of scoping or retrying or ensembling changes that. It can do the recon, and once a human aims it at the class and surface, one of them can even craft and fix an exploit. The lever that would make it genuinely useful, fine-tuning, is real and proven and needs hardware I do not have.
A weapon that only fires when a human has already aimed it is not much of a weapon yet. But it is an honest result, it is pointed in the right direction, and it cost me exactly the right amount of sleep to learn to read the map before walking the wall. If you take one thing from this series, let it be that: the negative result was always available in the literature, and the next real step has a price tag measured in GPUs, not in nights.