Are Proprietary Cyber Risk Quantification Models Black Boxes?

Buyers evaluating cyber risk quantification run into the question early. Proprietary models are black boxes, the argument goes, and a model built on an open framework is not.

‍

The question is a reasonable one. The test attached to it does not work, and a buyer applying it will end up trusting some platforms they should question and questioning some they should trust.

‍

What Does Proprietary Describe?

‍

Ownership of the method, and nothing else.

‍

A proprietary model is one its vendor developed rather than assembled from a published specification. That is a statement about where the method came from. It carries no information about whether anyone outside the vendor can examine the figures the method produced, which is the thing the word black box is reaching for.

‍

Two platforms can both describe themselves as proprietary and sit at opposite ends of what a buyer can inspect. So can two platforms built on the same open framework.

‍

Why the Framework Test Fails

‍

Because an open framework is a published way of structuring an estimate, not a window into a calculation.

‍

Open frameworks set out how to decompose risk into factors and how to combine them. Anyone can read the specification, which is a real benefit and the reason they are widely used. What the specification cannot tell a buyer is what numbers went into one particular analysis, where those numbers came from, or whether the resulting figure can be followed back to anything.

‍

A platform can implement an open framework faithfully and still hand a board a number with no route back to the inputs behind it. Declaring the method is not the same as exposing the evidence.

‍

Documentation and Evidence Do Different Jobs

‍

A vendor satisfies the documentation standard by publishing a methodology nobody has time to audit. That is the easier of the two to supply, and it is what the framework question tests for.

‍

Evidence is the harder one. It means the individual records underneath a figure, available to somebody who wants to check rather than described to somebody who will not.

‍

A Test That Separates Them

‍

Three questions, each answerable during an evaluation rather than after signing.

‍

One, Can You See the Events Behind the Figure?

‍

Not the methodology and not a sample. The individual simulated events the model generated for this company, each with its type, its initial vector, the assets it reached and the loss it produced. A platform whose output stops at a chart has not answered this.

‍

Two, Can You Watch a Headline Metric Resolve Against Those Rows?

‍

A figure that can be decomposed into the operations that produced it, performed against the real table rather than an illustration. An average annual loss is a sum of the loss across every simulated event divided by the number of years simulated. A buyer should be able to see that happen rather than be told it happened.

‍

Three, Can You Reproduce the Figure Outside the Tool?

‍

The strongest of the three, because it needs nothing from the vendor. Export the event set, open it in a spreadsheet, and arrive at the same number the platform reported. If that works, the arithmetic is confirmed independently and nothing has to be taken on the vendor's word.

‍

How Kovrr Answers Them

‍

With the same event set the model has always produced, now sitting in front of the user rather than only in an export.

‍

Every Kovrr quantification generates one row per simulated event, and that catalog has been available to customers for as long as the model has produced one. What changed this year is where it lives. The Event Catalog puts the table inside the CRQ platform as a tab in every quantification, with the full download still there.

‍

It also answers the second question directly. Selecting a headline metric reorders the table so the columns taking part move into view, then walks the calculation one operation at a time against the live rows. Average annual loss resolves into selecting the net loss column, summing it across every event, and dividing by the simulated years including those in which nothing happened. The 1-in-100 figure totals each simulated year, ranks all of them, and marks the value at the worst one percent.

‍

The third question is a spreadsheet exercise. Download the catalog, sum the loss column, divide by the number of simulated years, and compare.

‍

Where the Inputs Sit

‍

The data feeding the simulation is proprietary, including insurance claims reflecting losses that were paid rather than losses that were announced. That is the part a buyer cannot inspect, and it is the part worth having, since a public dataset would carry less information and every competitor could use it.

‍

Nothing downstream of those inputs is hidden. How the model turns that data into a company's own figures sets out the full path. A buyer does not have to accept the inputs on faith to verify the output.

‍

What Should a Buyer Ask in an Evaluation?

‍

Four requests, all of which a platform either satisfies in the session or does not.

‍

• Show me the event set: the simulated events behind the headline figure, listed and readable, for the company being evaluated.
• Decompose one metric: against the real rows rather than a worked example in a document.
• Export it: and confirm the figure reproduces outside the platform.
• Name the data categories: what kinds of source the model draws on, and which of them are proprietary.

‍

A platform that satisfies the first three is checkable whatever its method is called. One that satisfies none is not, whichever framework appears on its website.

‍

The Question Behind the Question

‍

Proprietary describes who built the method. Open describes who published it. Neither says whether a figure can be followed back to the events that produced it, and that is what a board member, an auditor or an actuary is asking when they use the phrase black box. The test that resolves it has three parts. Whether the simulated events are visible, whether a headline metric can be watched resolving against them, and whether the figure reproduces outside the tool. Kovrr's event catalog has always been downloadable and now sits inside the platform with the arithmetic traced against it, so all three can be answered in an evaluation rather than argued about afterward.

‍

To run those three questions against a live quantification, book a demo with our cyber risk experts.