Case study / StackAttest
Making production readiness something you can check
The engineering problem was not gathering evidence. It was designing an output that stays honest under commercial pressure, including the pressure to make the number look better.
What did TechSteps build with StackAttest?
A software validation product that collects evidence from a live application, its source, its dependency graph and an isolated runtime, maps findings to named security and engineering standards, and produces a Passport the publisher controls. The defining design decision was keeping the deterministic score strictly separate from AI model review, so consensus can never move a measurement.
Snapshot
- Product
- StackAttest
- Publisher
- Tech Steps LLC
- Category
- Software assurance
- Evidence sources
- Four, independent
- Standards mapped
- ASVS, SSDF, CWE, API Top 10, WSTG, SLSA
- Output
- Publisher-controlled Passport
Context
What existed before.
A working demo, a passing test suite and a live URL are all compatible with software that has weak access control, vulnerable dependencies, no recovery path and no observability. The signals people use to judge readiness measure something adjacent to it.
That gap widened as more software began to be assembled quickly. The rate of producing something that runs rose faster than the rate of establishing that it should be trusted.
The people who need to make that judgement are usually not able to read the codebase. A founder deciding whether to launch, an accelerator reviewing a portfolio, a buyer running diligence. They need evidence with a stated scope, not a badge.
Constraints
What made it difficult.
Without these, the problem would have been ordinary. They are the reason the design looks the way it does.
- 01
The output has to survive being wanted
The subject of the validation is also the customer. Any design where commercial pressure can move the score is a design that will eventually produce a meaningless score. The architecture has to make that structurally difficult rather than rely on good intentions.
- 02
Model output is probabilistic and the score cannot be
AI review adds genuine value in reading code and spotting patterns. It also varies between runs. A measurement that moves when nothing about the software changed is not a measurement.
- 03
A single number hides the thing that matters most
A high score on shallow evidence and a high score on thorough evidence mean completely different things, and compressing them into one figure destroys exactly the information a serious reader needs.
- 04
Access has to be bounded
Examining someone's source and running their application requires access they are right to be cautious about granting. The permission boundary is a product constraint, not an implementation detail.
- 05
It must not be mistaken for a compliance certification
Ambiguity here would be commercially convenient and dishonest. The product has to state what it is not, prominently, in a market where competitors benefit from vagueness.
System map
The parts that had to trust each other.
- 01
Live surface
The running application as an ordinary client reaches it, tested from outside.
- 02
Source
The code examined for patterns that map to named weakness classes.
- 03
Dependency graph
What the application pulls in, where much of the real risk lives and where teams have least visibility.
- 04
Isolated runtime
The application executed in a controlled environment, because some properties only appear at runtime.
- 05
Deterministic scoring
Evidence mapped to standards and scored reproducibly. The same evidence always produces the same number.
- 06
Model review council
Independent reviewers reporting agreement, objection and minority opinion. Deliberately has no path back into the score.
The important structural property is the separation on the right: evidence flows into a deterministic score, and model review runs alongside it without an edge back into the number.
Decisions
What we considered and what we chose.
A decision with no rejected alternative was not a decision. These are the forks where a different answer would have produced a different product.
| Decision | Alternatives considered | What we chose and why |
|---|---|---|
| How to express findings | A proprietary rating, or mapping to named public standards | Named standards: OWASP ASVS 5, NIST SSDF, CWE, OWASP API Security Top 10, WSTG and SLSA. A private scoring system is unauditable by design, and mapping to public controls means a reader can check what the control actually requires. |
| Role of AI review in the score | Let model consensus adjust the score, or keep them strictly separate | Strictly separate. The score is computed deterministically from evidence. Model review reports alongside it. Consensus never changes the number, which is what makes the number reproducible. |
| One reviewer or several | A single strong model, or multiple independent reviewers | Multiple, because a single model has a consistent set of blind spots. Disagreement between reviewers is information, and reporting objections and minority opinions is more useful than a smoothed consensus. |
| How to report the result | A single headline score, or several separate signals | Four signals reported separately: score, evidence coverage, validation confidence and model consensus. This is commercially worse and epistemically much better, because it shows how much was actually examined. |
| Binary result or graded verification | Pass or fail, or a verification ladder | A ladder from registration through external, code, runtime, production and continuous validation. Readiness is not binary, and a badge would flatten a real distinction while encouraging overclaiming. |
| Who controls publication | Automatic publication, or publisher control | Publisher control with the evidence trail intact. Validation nobody can withhold is validation nobody runs voluntarily on their own product before they are forced to. |
Implementation
How it was actually built.
- 01
Evidence collection
Four independent sources built first, because each answers questions the others cannot. Reading source tells you nothing about what a deployed instance actually exposes.
- 02
Control mapping
Findings mapped to named standards with the mapping itself made explicit, so a reader can follow a result back to the control it relates to.
- 03
Deterministic scoring
Scoring built as a pure function of collected evidence, before any model review existed, so there was never a code path where the two could become entangled.
- 04
Model review layer
Independent reviewers added alongside the score with no write path into it. Their output is agreement, objection and minority opinion, reported as its own signal.
- 05
Verification ladder and grading
Levels from registration through continuous validation, with each verified capability carrying an evidence grade describing how it was established.
- 06
Passport and boundaries
The published artifact, with the SOC 2 distinction stated on the product itself rather than buried, because that boundary is the thing most likely to be misread.
Verification
How we checked it worked.
- Scoring confirmed reproducible: identical evidence produces an identical score
- No code path from model consensus into the deterministic score
- Control mappings checked against the published standards they reference
- Evidence grades traceable to the method that established each capability
- Run against our own security product, producing a public Passport anyone can read
- Boundary language reviewed so the SOC 2 distinction is stated rather than implied
Outcome
What we can and cannot claim.
Every statement below is labelled with where it comes from. Nothing here is an unattributed percentage.
- Third-party verified
The output is public and checkable on a real product
The SecAI Passport is published at https://stackattest.com/p/secai-22dd62 and shows verified capabilities, evidence grades and the levels not yet reached. A reader can judge the output without taking our description of it.
- Design property
A score cannot be moved by model opinion
Scoring is a deterministic function of collected evidence with no write path from the review layer. This is structural rather than a policy, which is the point.
- Design property
Coverage is published alongside the result
Evidence coverage and validation confidence are reported as separate signals, so a strong score on thin evidence is visibly distinguishable from a strong score on thorough evidence.
- Observed by us
It changed how we write about our own work
Building an assurance product made unsourced claims uncomfortable to write. That is why case studies on this site label where each statement comes from. Our own observation, offered as such.
What stayed out of scope
- StackAttest is not a compliance certification and does not make an organization compliant with anything.
- It does not replace SOC 2, ISO 27001 or any organizational audit. It examines software, not the organization around it.
- It is not a penetration test. Automated validation and an expert attempting to exploit a system are different activities.
- A validated application can still contain an undiscovered vulnerability. That is precisely why coverage is published next to the score.
Lessons that changed how we work
-
Put the boundary between deterministic and probabilistic in the architecture
A policy that model output will not affect the score is a promise. No code path from one to the other is a property. We now apply that distinction to every AI system we build for clients.
-
Publishing coverage is what makes a score meaningful
The number alone invites the wrong conclusion. Reporting what was not examined turns a marketing artifact into evidence.
-
Disagreement is a finding
Where independent reviewers objected is often more informative than where they agreed. Averaging that away discards the most useful signal in the process.
-
State the boundary before someone else tests it
Saying plainly that this complements SOC 2 rather than replacing it costs a little commercially and removes the failure mode where a customer discovers the limit at the worst moment.
Have something with constraints like these?
The interesting work usually arrives as a set of things that cannot all be true at once. Tell us what you are trying to make work.