I spent a day packaging an internal security toolkit — a few dozen markdown playbooks — into something publishable. The playbooks were written during real engagements, so before anything could ship I needed a gate that guaranteed no client name, host or credential survived into the public artifact.
I wrote that gate. I validated it against a known positive and a known negative. It found
two real leaks on the first run, which felt like proof it worked. It said PASS —
clean, 77 files.
It was wrong five separate times. Four of those were my fault, and every single failure had the same shape.
The shape
Every one of these failures is an instrument that cannot distinguish two states — and when it can't, it reports the one that requires no action. Not randomly. Always in the direction that lets you keep going.
1. A single null byte hid an entire file
One playbook documented null-byte injection and contained a literal 0x00. That
makes grep treat the file as binary, so it silently skips it. My gate iterated 77
files, scanned 76, and reported a clean pass on all 77.
When I fixed it — refusing to scan what can't be read, rather than trusting the count — the run immediately surfaced seven client-name leaks that had been sitting inside that one invisible file the whole time.
The gate could not tell no leaks in this file from this file was never opened. Both look like silence.
2. The allowlist cleared whole lines
Allowlists are how you stop a gate crying wolf over teaching placeholders like
target.com. Mine matched an allowed pattern and then dropped the entire line.
So a line reading "...YOUR NOTES on the <client> engagement..." was cleared
— because notes was allowlisted. The client name rode out on the same line as the
exemption. Allowlist the token, never the line.
3. gitleaks scanned history, not the files
Not my bug, but my misuse. gitleaks detect --source <dir> on a git repo
reads commit history. It does not read your working tree.
I planted an AWS key and a GitHub PAT in a tracked file and ran the gate. It reported
PASS. The secrets were real, present, and about to be published — and the tool
was working exactly as documented, just answering a question I hadn't asked. You need
gitleaks dir and gitleaks git; either alone is a confident
half-answer.
4. Every gate scanned the product. Nobody scanned the repo.
This is the one that would have actually hurt.
All my gates ran against the build output. They passed. But the thing that reaches GitHub is the repository, and the repository contained the gate's own configuration — including the deny list the gate uses to recognise client names. That file was, by construction, a complete list of every client the toolkit had ever been used on.
The leak-prevention machinery was the largest leak in the project. A commit message named two clients as well, and commit messages are permanent.
The instrument measured the artifact. I concluded the publication was safe. Those are two different sentences, and I never noticed I'd swapped one for the other.
5. The test harness was broken, so I nearly deleted a working gate
Having built a repo scanner, I tested it: plant a client name, confirm it fails. It passed. I spent twenty minutes convinced the scanner was broken.
The scanner was fine. My test picked the probe name with grep -vE '^#', which
strips comments but not blank lines — so the probe was the empty string and I had
planted nothing at all. I was one step from "fixing" a gate that already worked.
The lesson isn't "write better tests". It's that a validator is an instrument too, and it fails the same flattering way. That test now asserts its own probe is non-empty before it will trust any result it produces.
The question that finds all five
There's one question that catches every failure above, and it takes about ten seconds:
"What two situations would produce this identical output?" If one of them is fine and the other is act now, the instrument hasn't answered the question. Go read ground truth.
Applied to the five: clean file vs unread file. Allowed line vs leak beside an allowed word. No secrets vs wrong corpus scanned. Safe to publish vs safe artifact, unsafe repo. Gate broken vs test broken. Every one is a pair, and every one defaults to the reassuring half.
What actually fixes it
Three things, in order of how much they buy you.
Prove the gate fails. A gate that has never failed on a planted positive is an untested gate, and its pass means nothing. Every gate I ship now has a self-test that plants a known leak and requires the gate to catch it — plus a vacuous-pass guard, because zero files scanned must never be reported as clean.
Scan what actually ships. Not what you built. Not what you intended to publish. Enumerate the bytes that leave the building, and scan those.
Write gates that fail in opposite directions. One gate proved nothing private survived. A second proved the document survived — because over-aggressive redaction had quietly turned "search engine" into "the target engine". The first gate loved that output. Two gates disagreeing is how you find the thing neither would catch alone.
Why I'm writing this down
Because I audit other people's controls for a living, and four of these five were mine — committed while being rigorous about someone else's output and casual about my own inputs. That asymmetry is the most reliable way I know to manufacture a false result.
The toolkit is open source and MIT licensed — github.com/krishnextgencyber/hackz-huntkit — offensive-security playbooks for AI coding agents, with the publication gates described above baked into its release process.
It shipped a day later than planned, with eight gates instead of one, and with a written list of what is verified by execution versus what is merely format-checked — because "it parses" is not "the tool used it", and pretending otherwise is just another instrument that can't tell two states apart.