Open any compliance dashboard and the rows are green. Ask what turned a row green and you usually get one of three answers: a policy document says so, somebody uploaded a screenshot in March, or a query ran against the live environment and returned a value on a date. Only the third is a measurement. The first two are claims about intent, and intent is not configuration.
This gap is not fraud and rarely even carelessness. It is how most compliance tooling is built. A framework requirement is satisfied by attaching an artifact, and an artifact is whatever the person mapping the control decided was close enough.
Three Things That Look Identical in a Report
A policy. "All administrative access requires multi-factor authentication." True as a statement of what the company wants. It says nothing about the four service principals excluded from the Conditional Access policy last quarter for a migration that finished.
A screenshot. A portal blade captured on a date, cropped to the setting that passes. It proves the setting was correct in that pane, at that moment, in that subscription. Configuration drifts daily, and a screenshot cannot tell you it has gone stale.
A check. A named query against the live API that returns a value: how many accounts hold standing Global Administrator, whether the storage account allows public blob access, whether the backup vault has an immutability lock and who can remove it. The value decides the row, and the row carries the date it was read.
A report that renders all three the same way is not telling you what you think it is telling you.
Point in Time Is Not the Same as Continuous
Even a real check proves state on a date, not that a control operated all period. That is exactly the distinction an auditor draws between design and operating effectiveness, and it is why what auditors actually test is a series of readings rather than one clean one. A single verified reading beats a screenshot. A repeated verified reading with a history is what a control owner can actually defend.
The other half of honesty is what happens when a check cannot conclude: a missing permission, a licence tier that does not expose the data, a resource outside the connected scope. If that renders as a pass, the report is worse than having no report, because a silent failure painted green looks exactly like a real answer. The three honest states are verified, attested, and unreadable.
What Checking the Real State Looks Like
SecValley scans read-only across seven surfaces, Microsoft 365, Entra ID, Azure, AWS, Google Cloud, Google Workspace, and GitHub, and run 1,103 configuration checks that return values rather than opinions. Each control opens to show what was read, in which subscription or tenant, and on what date, and each one carries the framework requirements it satisfies, from CIS Benchmarks to SOC 2 criteria, so a green row can be traced back to the query that produced it. A control that maps to no requirement in your active frameworks says so on its face instead of quietly inflating a coverage number, and a check that could not be read is reported as unanswered, never as passed.
The point is not a higher score. It is that every row in the report can answer one question: what ran, what did it return, and when.
The One Question to Ask Your Current Tool
Pick a green row, any row, and ask it to show the value it read and the timestamp. If the answer is a document or an upload, that row is an assertion. It may well be a true one. It is just not a check.
Frequently Asked Questions
Is a policy document worthless as compliance evidence?
No. Policies, contracts, training records, and tabletop exercises are legitimate evidence and no scanner will ever produce them. They only have to be labelled as assertions, so nobody later mistakes a stated intention for a measured state.
Does passing a CIS Benchmark mean we are secure?
It means the configurations in that benchmark matched their recommended values when they were last read. It is a floor with a date on it, not a verdict on your security programme, and controls outside the benchmark are simply not in scope of the answer.
How often should configuration be re-checked?
Often enough that the answer is younger than the rate of change in your estate, which for most cloud tenants means continuously rather than quarterly. We wrote up the reasoning in how often you should scan your cloud environment.
Can every framework requirement be checked automatically?
No, and a tool claiming otherwise is describing something else. Identity, privilege, logging, network, and storage configuration is largely queryable. Governance, training, and vendor management stay attestations with a named owner and a date.
