A cockpit for the red-team runs
Part of my EVA series, where I build the cockpit I pilot GORED runs from. GORED is the harness that points small local language models at my own web apps and measures, honestly, whether they can break them; EVA is the web surface I drive it from. The harness and the cockpit both live in t0ul/GORED, and these write-ups live in t0ul/ai-security-engineering. Every run EVA shows you is a kept, hash-verified trace.
Intro
I spent a few weeks on a question I wrote about in my GORED series: can a small language model I run on my own laptop find the security bugs in an app I wrote? The harness that answers it works. It points a local model at a target, lets it recon and probe, and an oracle decides whether anything was actually captured by reading the real server’s bytes. That part was done.
What was not done was my ability to sit with it. Every run was a command with a fistful of flags, a wall of scrolling output, and, at the end, a JSON trace file on disk. If I wanted to know which of two models did better, I grepped. If I wanted to see what the model had been thinking when it went down a dead end, I grepped. I was running real experiments and reading them like a server log, which is to say badly, at the wrong hour, and with no feel for the shape of what had happened.
So I built EVA. It is a web cockpit I launch a run from and watch land, and this post is about why it exists and the single decision that shaped the whole thing. I will save the interesting engineering, the part where one optional hook makes an interactive run and a headless CI run the same loop, for the next post. This one is the frame.
EVA, the cockpit I built to fly all of this. The preflight panel on the left is the whole argument in miniature: the oracle reads real bytes, the egress gate is armed, and the model is the thing being tested, not the thing being trusted.
I. The run I could not watch
Here is what a session looked like before. I would pick a target, pick a model, pick a stage, and type something like a dozen flags into a sweep binary. It would boot a local model, which takes a few seconds, and then it would start talking: every HTTP request, every tool call, every model reply, every observation it wrote to its blackboard, all of it streaming past faster than I could read.
When it finished I had two things. I had an attack-success number, which told me whether a capture happened but nothing about how. And I had a trace file, a hash-chained record of every step keyed by one trace id, which told me everything about how but only if I was willing to read JSON lines by hand. Both of those are good things to have. Neither of them is something you can reason about while it is happening.
The trace being hash-chained matters here, and it is worth a sentence because it is the reason EVA can be trusted later. Every record links to the one before it with a hash, so if I, or anything else, edited a run after the fact, the chain breaks and a verify step catches it. That is the property that lets me replay a run weeks later and know I am looking at what really happened, not at a story I told myself. The cockpit leans on it constantly. But a property you can only inspect with a text editor is not doing you much good in the moment, and the moment, watching a model decide, is exactly where the learning is.
II. One engine, two front doors
This is the decision everything else hangs on, so I want to be plain about it.
EVA is not a new program that runs pentests. It is a thin web layer over the exact same library the command line already drives. The engine lives in the harness’s internal packages: the function that runs a sweep, the probes that do the actual poking, the validator that fuses and ranks findings. The command line calls that engine headless. My CI calls that engine headless. EVA calls the same engine in-process and puts a window on it. Nothing shells out, and there is no second copy of the logic that could quietly drift from the first.
I held that line on purpose, because the alternative is the thing that makes tools like this lie. If the interactive version and the automated version are two different code paths, then the run you watch in the browser and the run your build pipeline uses as a gate are two different runs, and the pretty one in the browser is the one you will believe. By making EVA a front door onto the same engine, what I watch is byte-for-byte what the gate runs. The window does not change the thing it is looking at.
In practice the cockpit is one small binary. You point it at a directory for its traces and it serves:
go run ./cmd/gored-eva -traces ./runs
# EVA cockpit on http://127.0.0.1:4100
That is the whole startup. It opens a SQLite file next to the traces for its run registry, so the log of what you have launched survives a restart, and it serves the cockpit on a local port. No cloud, no account, nothing leaves the box, which has been the rule for this whole project from the first day.
III. What the cockpit actually shows
Once it is up, a run is a form instead of a flag soup. You pick the target, the stage or flow, the model, how many trials, and two knobs I will come back to in later posts: an autonomy dial and the prompt the stage runs with, editable right there and recorded with the run so it stays reproducible.
The preflight panel in the screenshot above is the part I am proudest of, because it is the argument of the whole project rendered as a status light. Before a run spends a single token it shows you three things are true. The oracle is wired to read the target’s real responses, so a capture means the server actually disclosed something, not that the model said it did. The egress gate is armed, so the contained target cannot call home; the harness even fires a deliberate beacon at a non-target host and shows you it was denied, right there in the trace. And the model is loaded as the subject of the test, the thing on trial, never the judge of its own work.
Then you launch, and instead of a scrolling wall you get a live page: the phase it is in, the blackboard filling up with what it has found, and, at the end, a capture verdict that came from the oracle and not from the model’s own claim. I will show that live feed properly in its own post, because the way it streams without the engine knowing a browser exists is a nice little piece of plumbing. For now the point is smaller and bigger at once: I can watch.
IV. LLM proposes, human disposes, on a dial
There is one more idea baked into the cockpit from the start, and it is the reason it is a cockpit and not a dashboard. A small local model, in my experience, is good at breadth and bad at judgment. It is genuinely strong at reading an app, naming candidate paths, drafting payloads, synthesizing recon. It is weak at the one thing that matters most, which is knowing what is worth attacking in this app. So EVA is built so a human can supply that judgment at the exact moment it is scarce, after recon and before the run spends its budget chasing a guess.
But I did not want to force that. So autonomy is a dial, not a switch. Turned down, the cockpit pauses at each gate and waits for me to pick or hint. In the middle, the model proposes and picks a default and carries on unless I step in. Turned all the way up, affectionately called berserk, it never pauses and runs exactly like the headless loop, which is the setting my CI uses. Same engine, same run, one knob deciding who picks the next move. What never moves is who confirms the finding: that is always the oracle, on the real bytes, at every setting of the dial.
How that dial is actually one optional hook in the engine, and why that hook was the difference between a clean design and a forked mess, is the next post.
Conclusion
That is the frame. EVA is a thin web cockpit over the same red-team engine my command line and my CI already run, built so I can launch a run, watch it think, and trust the recording afterward because the chain cannot be quietly rewritten. It did not make the models better, and the GORED series is honest about where they top out. What it did was turn a pile of sweeps I could only grep into something I could actually fly and reason about, which is the difference between running experiments and learning from them.
Next I will open up the one piece of engineering I am happiest with: a single optional checkpoint in the engine that lets a human-piloted run and a fully autonomous CI run be the very same loop, with no second code path and no frontend logic leaking into the core. If you have ever wanted an interactive mode and a headless mode without maintaining two of everything, that post is for you.