What Happens When a Cyber Risk Number Meets a Skeptic

TL;DR

‍

• Boards have stopped arguing about whether cyber risk belongs in financial terms, and started asking how each figure was derived.
• The person who has to answer is the security leader presenting the slide, never the vendor who built the model.
• Property catastrophe insurance has already run this argument in public, where regulators and consumer advocates are demanding access to model weights and output distributions before a model can be used to set a rate.
• The counterargument has force. Publishing fifty thousand rows of model output can bury a question instead of answering it.
• What settles the question is a traceable path from any headline figure down to the records that produced it, in the way an auditor follows a financial statement into the accounts beneath it.

‍

The Growing Need to Justify the CRQ Numbers

‍

The meeting goes fine until someone asks where the figure came from.

‍

A security leader presents an annual loss estimate to the board. The audit committee chair, twenty years in financial services, wants to know how it was derived. In most organizations, the honest answer is that a vendor's model produced it. The answer closes the conversation in a way nobody intended, because the figure stops working as evidence and starts working as an assertion.

‍

Cyber risk quantification has spent a decade arguing that financial estimates beat color-coded heat maps, and that argument has largely been won. Boards ask for dollars now. Regulators ask for dollars. What remains unresolved is the question arriving immediately after the dollars, and the people who have to field it are rarely the people who built the model.

‍

The Burden of Proof Sits With Whoever Is Holding the Slide

‍

A vendor builds the model and stands behind it, yet the vendor sits outside the room when the audit committee asks its question. The security leader sits inside it. Whatever confidence the model deserves has to travel through a person who did not build it, toward an audience with no particular reason to extend credit, often under time pressure and with a budget request attached to the outcome.

‍

The position inverts something that normally works in a security leader's favor. On their own subject, they are the best-informed person present, and that standing is what makes board education possible at all. Handing them a figure they cannot decompose takes the standing away, because an appeal to advanced algorithms and proprietary methods makes the presenter sound less informed than they were before the tool arrived. Nobody can teach what they cannot explain, and teaching is most of what board reporting consists of.

‍

Property Insurance Has Already Had This Argument in Public

‍

Cyber quantification tends to discuss transparency as a preference. One industry further along the same road has been treating it as a legal question for years.

‍

California has spent the past two years fighting over whether insurers may set wildfire premiums using catastrophe models that nobody outside the modeling firm can inspect. Consumer Watchdog's testimony to the state's modeling workshop asked for access to the model itself, along with its weights, its training data and its output distributions, on the ground that without those things nobody can verify either the math or the science behind a rate. The testimony notes that small changes to model weights become highly leveraged, so modest adjustments nobody can see produce large swings in the number a homeowner pays.

‍

One passage in that testimony should stop any cyber risk practitioner reading it. Models built from similar literature, the group reports a former industry executive saying, can produce "average annual losses 2 or 3 times different than each other." Average annual loss is the same metric cyber quantification platforms put at the top of the dashboard, computed the same way, and the observation that two defensible models can differ on it by a factor of three is not a reason to abandon the metric. It is a reason to insist on seeing which events produced it.

‍

Writing in CalMatters, climate financial policy strategist Jordan Haedtler made the related point that outsiders have little information about the inputs feeding private models, which leaves regulators without an independent benchmark against which to test a rate increase. The remedy under discussion in California is a publicly built model, which is a heavy answer to the problem. Heavy answers get proposed when lighter ones have been refused for long enough.

‍

Cyber is arriving at the same junction from a different direction. The SEC's disclosure rule, DORA and NIS2 all push organizations toward demonstrating how a conclusion about exposure was reached, with documentary support behind it. A figure that survives a friendly boardroom will eventually meet an examiner who has more formality available and less patience for an appeal to proprietary methods.

‍

An Unexplainable Figure Costs More Than No Figure

‍

Follow a quantified estimate that cannot be traced through its short life. It gets presented once, a director asks how it was produced, and the answer gestures at Monte Carlo simulation and proprietary data. The director discounts it, reasonably, because an unverifiable number supplied by an interested party is worth less to them than their own judgment of the same situation. Nothing about that exchange is hostile. The director is doing the job a director is supposed to do, which is to weigh evidence by how well it can be examined, and the figure simply fails that test.

‍

The damage spreads past the one meeting. Budget requests anchored to the figure inherit its credibility and weaken when it does, so a security program loses the argument it was built to win. Second-line risk functions cannot challenge a model they are unable to evaluate, which leaves an organization's own reviewers without a way to perform the independent challenge their charter demands. Internal audit inherits the same problem on a longer timeline, and by then the figure has been repeated in enough places that walking it back carries its own cost.

‍

Opening a Model Can Be Its Own Kind of Evasion

‍

The counterargument deserves a hearing, because it lands harder than most vendors admit.

‍

Transparency can function as a way of burying a question instead of answering it. A model that responds to "where did this come from" by producing fifty thousand rows has explained nothing. It has relocated the work onto the person who asked, who now needs time and quantitative training they may not have in order to judge whether the answer holds. A document nobody can audit inside the available time is less defensible than a summary, with the added insult of appearing cooperative while being useless.

‍

A second version of openness makes decisions harder instead of easier. If every assumption inside a model is exposed and contestable, a risk committee can spend its hour litigating parameters instead of deciding what to fund. Some organizations have watched that happen and concluded, sensibly enough, that a defensible figure they can act on beats a transparent one they cannot stop arguing about. Two models disagreeing on the same company is a familiar version of the same trap.

‍

Both objections point at one missing piece. Volume is useless without a path through it. Handing someone the underlying data answers a question nobody asked, since the question was never about what data a vendor holds. The question was how this particular figure, the one on the slide, came out of that data.

‍

Traceability Is What Makes Volume Usable

‍

An auditor reviewing a financial statement does not read the general ledger end to end. They follow a number from the statement down into the accounts that produced it, test a sample of those accounts, then form a view. What makes the process work is the path, which exists, is documented, and can be walked by someone who did not keep the books.

‍

Risk models have mostly lacked an equivalent. The inputs get described in a methodology document, the outputs appear on a dashboard, and the connection between the two is asserted. A security leader presenting an average annual loss figure can say what it means, can describe in general terms what feeds the model, and has no way to show the arithmetic that turned one into the other.

‍

The components of that path are not exotic. A model that simulates events already produces a record of every event it simulated, with a cost attached to each one. Every headline metric sitting on top of those records is arithmetic over them, and arithmetic can be displayed. An average is a sum divided by a count. A one-in-a-hundred-year figure is a sorted list with a marker at the first percentile. None of the operations involved are difficult to follow once someone agrees to show them, which is the part that has been absent.

‍

What has been missing is commercial motivation, since a vendor whose model holds up loses nothing by opening it, while a vendor whose model does not hold up loses everything. Opening a model is therefore a statement about the model before it is a statement about transparency. The firms that can afford the move are the ones whose event data came from somewhere defensible, and the firms that cannot have been content to let the category keep its reputation for opacity, since that reputation protects them.

‍

California is working toward its answer through legislation, which takes years and produces a model built by committee. Cyber has the option of getting there faster, because the records already exist inside every simulation that has ever been run.

‍

Ready to see how a quantified figure traces back to the events behind it? Schedule a demo and walk through your own numbers with a Kovrr analyst.