The Risk Rancher Field Manual, Ch. 5: The Deep Dive
Part of The Risk Rancher Field Manual, a hands-on guide to starting, scaling, and running a vulnerability management program.
A scanner casts a wide, shallow net. A pentester dives deep in one spot. A pentest, a penetration test, is a skilled person rather than a program, trying to break into one target for a fixed time and writing up what they found. Both have their place, but they are not substitutes, and the single most common mistake in this whole field is paying for the second when you have not done the first. This chapter is when a pentest is worth paying for, which kind, and how to grade the report you get back.
Do you even need one yet?
Here is the honest gate, the one no pentest firm will put on their sales page: if you are not already patching the obvious, a pentest is a waste of money. A tester will find the same out-of-date software your scanner found, add a few things, and hand you a nicer-looking PDF. A typical small engagement is a couple of weeks of one tester’s time, a five-figure sum, and you will have spent it to confirm the backlog you already had.
The value of a pentest is not in finding the easy stuff. It is in finding the things automation physically cannot: the business-logic flaw where a user can skip the payment step, the chain where three low-severity issues combine into one critical, the creative human move a scanner has no concept of. You only get that value if the easy stuff is already handled, because otherwise the tester spends the engagement, and your money, rediscovering it.
So earn the pentest. Do your scanning and your patching first. Then bring in a human to find what the machines miss.
Two different nets. One covers the estate; the other goes all the way down on one target. Neither replaces the other.
When is it actually time? When a customer or regulator requires one and a deal depends on it. When you are shipping something genuinely new and high-stakes. Or when you have wrung out everything the scanners can find and you want a skilled adversary to tell you what you cannot see about your own system.
The types
“Pentest” covers a range, and buying the wrong one is its own kind of waste.
- Web application and API tests are what most companies actually need: a skilled human attacking your app the way a real attacker would.
- Network and infrastructure tests probe your perimeter and internal segmentation.
- Cloud tests go after your cloud configuration and identity, the modern soft underbelly.
- Red team engagements are goal-based and stealthy: can we get to the crown jewels without being caught, across the whole organisation, over weeks? These are powerful and expensive, and you do not need one to check a compliance box.
Buy the test that matches your real risk, not the most impressive-sounding one. A payments company needs a hard look at its payments app far more than it needs a cinematic red team.
How to scope it so you don’t waste the money
The difference between a useful pentest and an expensive scan is the scope and the tester. Control both.
Scope it to a real target with a real question. “Test everything” produces shallow coverage of a lot; “can an unauthenticated user reach another customer’s data in this app?” produces a sharp, useful answer. Give the testers access, documentation, and a test account, because a blind test of a complex system burns half the budget on reconnaissance you could have handed them on day one. Ask for a sample report before you sign; if their idea of a finding is a scanner’s output with a severity score pasted on top, walk away. You are paying for narrative, reproduction steps, and honest business impact. And insist on a retest after you fix, because a finding you never verified closed is a finding you never closed, which is the whole subject of chapter seven.
Try it. Grade a report before you pay for the next one. Run any pentest report you can get (your last one, or a vendor’s sample) against this checklist, one point each: (1) every finding has reproduction steps a developer could actually follow; (2) impact is written in business terms, not just a CVSS number; (3) at least one finding is something a scanner could not have found, a logic flaw, a chain, an abuse of intended behavior; (4) findings are specific to your system, not boilerplate; (5) there is a concrete, prioritized remediation per finding; (6) a retest after you fix is included in the price. Four or less out of six and you are looking at an expensive scan. Read the list the other direction and it is your scoping worksheet. The statement of work is the document that says what the tester will and will not do, for how much; demand these six in it and you will get a report worth feeding to the herd.
What you do with the report
A pentest report is not a separate universe, and it does not belong in a drawer. Those findings go into the exact same engine as everything else: triaged, prioritized, routed to an owner, fixed, and verified. The report is just another source of findings, a slower and more expensive one that happens to be much higher quality. Feed it to the herd like the rest.
If you are the pentester reading this, the tool in chapter ten was built with you in mind as much as the in-house team. It takes the report in: you upload it, write each finding up beside it as a draft, and promote the drafts into the same queue the client’s scanner findings sit in, so your work lands in the engine instead of dying in a Word document. It writes the report back out, branded, as Markdown or a page you print to PDF, from the findings you pick. And it makes no network calls, which matters when the findings are a client’s. Two words from that world you will meet again: the CVSS vector is the string behind the score, which says how the flaw is reached; the CWE is the catalog of kinds of weakness a CVE belongs to, such as injection.
That is the whole of buying a pentest well: earn it first, buy the type that matches your real risk, and grade the report against six points. Now you have the nets cast and, where it is warranted, a human in the water. Findings are coming in from every direction: scanners, pentests, bug reports, that one engineer who keeps emailing you things. Which brings us to the real work, the part the whole book has been walking toward and the part your spreadsheet cannot survive. Part 3: the vulnerability management engine. We start where everyone starts, drowning, and we learn to sort the herd.