Cyber Risk Quantification Vendor Evaluation Checklist

Buying a cyber risk quantification platform means asking a vendor to defend numbers that will end up in front of a board, an auditor, or a regulator. Most evaluation calls never press that hard. The questions below are the ones that separate platforms from each other, and they are written so that any vendor can answer them, including vendors who would beat Kovrr on several of the answers. A few of these questions have no vendor-side answer at all, because what they test is the state of the security organization doing the buying.

‍

A full evaluation covers five areas. Start with how the loss models get built and validated, then push on how much of the math the vendor will show, what the platform can and cannot model, what comes out the other end for boards and regulators, and what the thing costs to run once the sales team has moved on. Take the same list to three vendors, keep the answers side by side, and the differences stop being a matter of who presented better.

‍

Every Number a Platform Produces Inherits the Quality of Its Data

‍

Two vendors can run the same simulation method and land on loss figures that differ by an order of magnitude, because the inputs differ. Data questions come first in any evaluation, and they are the easiest place to spot a vendor who is reselling public breach reports with a pricing page attached.

‍

What Feeds the Loss Models, and How Current Is It

‍

Ask what categories of data sit behind the frequency estimates and the severity estimates separately, since vendors often have one and not the other. Frequency data tends to come from incident reporting, threat intelligence, vulnerability telemetry, and disclosure records. Severity data is harder to come by, because the cost of an incident is rarely disclosed in public, which pushes vendors toward insurance loss records, regulatory filings, litigation records, or published industry surveys. A vendor who cannot describe the severity side in any detail is estimating it, and the estimate deserves scrutiny.

‍

• Refresh cadence: Ask how often each data category updates and when the last update landed.
• Sample scale: Ask how many loss records sit behind the severity distributions, as a number.
• Relevance filtering: Ask how data gets narrowed to your industry, region, revenue band, and technology profile.

‍

How the Models Were Validated, and Against What

‍

Validation separates a statistical model from a calculator with confident formatting. Ask what the validation process involves, how often it runs, who performs it, and whether any of it is independent of the team that built the model. Backtesting against historical events, sensitivity testing on individual parameters, peer review of model changes, and benchmarking outputs against observed loss rates are all reasonable answers, and a vendor should be able to describe their process without retreating into the word proprietary. Ask also what population the validation ran against and how large it was.

‍

Whether Model Updates Arrive With the Subscription

‍

A platform that ships recalibrated models on a schedule differs from one that hands over a structured method and leaves the estimating to you. Both have a place, and the second costs considerably more internal time than the sales process tends to suggest. FAIR-based tooling often sits closer to the second category, since the standard describes how to decompose a risk without supplying the data to populate it. Ask directly how often the underlying distributions change, what triggers a change, who signs off on it, and whether updates are included or priced separately.

‍

A Number Nobody Can Explain Will Not Survive Its First Challenge

‍

The first serious test of a quantification program arrives when a CFO asks why the annual loss estimate dropped by 30 percent in a quarter when nothing visible changed. Vendors vary enormously here, and the variation rarely shows up in a demo, because demos run on static data. Ask for the explanation path before signing, since finding out later costs a credibility hit with the exact audience the program was bought to serve.

‍

Walk One Result Back to Its Inputs

‍

Pick a single output on the screen during the demo, a loss figure for one scenario, and ask the vendor to trace it backward through the model. The answer should reach specific inputs, including the controls assessment, the asset data, the firmographic profile, and the external data that drove the frequency. Vendors who answer this well will show the chain without preparation. Vendors who need to come back to you after the call are telling you something about how the model behaves under pressure.

‍

Find Out What Happens When the Number Moves

‍

Quantification results change between runs for several reasons, and the useful platforms distinguish between them. A change driven by a control upgrade means something different from a change driven by a model recalibration on the vendor's side, and a buyer needs to tell those apart to report honestly on program performance.

‍

• Attribution: Ask whether the platform records why each result differs from the previous run.
• Model versioning: Ask whether recalibrations are logged separately from customer-side input changes.
• History depth: Ask how far back the result history goes and whether old runs stay reproducible.

‍

Ask What an Auditor Gets to See

‍

Regulated buyers should ask what documentation exists for the methodology itself, who is allowed to read it, and whether it has been reviewed by anyone outside the vendor. Methods built on published standards carry an advantage here, since an auditor can read the standard independently. FAIR carries weight with audit and risk committees for exactly that reason, and a vendor claiming conformance to the Open Group's Open FAIR standard should be able to name the version. Ask also whether model documentation survives a vendor change, because a quantification program that cannot be defended after a contract ends was never defensible.

‍

What a Platform Cannot Model Shapes the Result More Than What It Can

‍

Every cyber risk quantification tool draws a boundary around the risks it will represent, and the boundary is almost never on the website. Scenario coverage questions find that boundary during evaluation instead of six months in, when a board member asks about a risk the platform has no way to express.

‍

Find Out Which Event Types the Models Treat Separately

‍

A ransomware event, a third-party service outage, a large data breach, and a run of small attritional incidents behave nothing alike in frequency or in cost distribution. Ask whether the vendor models these separately or runs one general loss model with different labels on the output. Neither answer disqualifies a vendor on its own, and a single well-calibrated model beats four badly calibrated ones. Ask what the modeling choice does to the tail, since that is where insurance decisions and board conversations end up living.

‍

Establish How Far Down the Analysis Goes

‍

Some platforms quantify the entity as a whole and stop. Others break results down to business units, facilities, asset groups, or individual applications, which matters for any buyer who needs to allocate budget across a portfolio instead of reporting one company-wide number. Ask what the smallest unit of analysis is, and ask what extra data collection that granularity demands. Top-down scenario analysis and bottom-up asset analysis answer different questions, and a buyer should know which one a platform is doing.

‍

Test What Happens to a Scenario the Vendor Never Anticipated

‍

Every organization carries a few risks that no vendor catalog contains, usually tied to a specific revenue process, an unusual data holding, a regulatory exposure, or a piece of infrastructure nobody else runs. Ask how a scenario like that enters the model and whether it joins the aggregate simulation or sits beside it in a separate report. Ask whether an existing risk register can be imported and quantified, and what format the import expects. The answers tend to be more honest when the question names a specific risk from your own environment.

‍

Reporting Is Where a Quantification Program Gets Judged

‍

The internal case for buying one of these platforms almost always rests on communication, with a board, an audit committee, a CFO weighing a budget request, or a regulator asking what the company knows about its own exposure. Ask to see the output itself instead of a description of it, and ask who assembles that output, since a report that takes an analyst four days to build each quarter has a running cost nobody quoted.

‍

Ask to See the Board Output, Not a Slide About It

‍

Request a board report pulled from a live customer environment with the names stripped, and read it the way a director would. Check whether every metric on the page explains itself without a presenter standing beside it, because the CISO will not always be in the room when the document gets read. Check whether the report connects exposure figures to specific decisions in front of the board, including budget requests and insurance limits. Ask how much of the report is generated and how much gets rebuilt by hand each cycle.

‍

Push on What the Platform Contributes to a Materiality Determination

‍

Public companies in the US have been living with the SEC's cybersecurity disclosure rules since 2023, and materiality determination has moved from a legal abstraction to something with a four-business-day clock attached. No platform makes that determination for you, and any vendor suggesting otherwise should be marked down. What a platform can supply is the loss distribution underneath the judgment, including the likelihood of exceeding a given financial threshold, an outage duration, or a record count. Ask how the thresholds get set and whether the method behind them is published.

‍

Find Out What the Platform Does for Budget and Insurance Decisions

‍

Two decisions justify most quantification purchases, which security investments to fund and how much cyber insurance to carry. Ask whether the platform models the exposure reduction from a specific control upgrade and what that estimate is calibrated against. Ask whether the loss curve can be read against policy limits, retentions, sublimits, and coinsurance, since an insurance conversation needs the whole distribution instead of an average. A vendor whose answer to both questions is one headline number is selling a dashboard.

‍

The Questions Where No Vendor Comes Out Looking Good

‍

Every section above has a strong answer available somewhere in the market. The three below do not, and they are the ones most likely to determine whether a quantification program survives its second year. Vendors tend to answer them in ranges and conditionals, which is appropriate, since the honest answer depends on what already runs inside the buying organization.

‍

Implementation Takes Longer Than the Sales Cycle Suggests

‍

Time to a first result and time to a result anyone will act on are different numbers, often by months. A first quantification can run on firmographic data and a controls self-assessment inside a day. Getting to results that survive scrutiny means mapping asset groups, reconciling the controls assessment with what is deployed, agreeing internally on what counts as a business unit, and getting data owners to sign off. Ask for the range across the vendor's customer base and ask what the slowest implementations had in common.

‍

• Data readiness: Ask which inputs the platform needs before the first run can happen at all.
• Internal hours: Ask how many hours the customer side typically spends during onboarding.
• Slow cases: Ask what caused the vendor's longest implementation to take as long as it did.

‍

Someone Has to Run This After the Contract Is Signed

‍

Platforms differ in how much quantification expertise they assume, and the assumption is rarely stated. Methods derived from FAIR benefit from an analyst who has been trained in the standard and can defend a decomposition under questioning. Automated platforms lower that requirement and move the expertise into model interpretation instead, which is a different skill and still a skill. Organizations without a risk analyst to spare should ask about managed quantification services early, because discovering the staffing requirement after purchase is how these programs stall.

‍

Integration Depth Varies More Than Any Vendor Will Volunteer

‍

Ask for named integrations with the systems already in place, including ServiceNow and whichever GRC platform holds the risk register today. Ask what the integration moves in each direction, since reading a risk register into the model and pushing quantified results back out are separate pieces of work. Ask what is a supported connector against what is an API and a services engagement. Vendor answers here get vague quickly, and specifics about field-level mapping tend to surface the difference.

‍

Commercial Terms Decide Whether the Program Can Grow

‍

Quantification purchases tend to start narrow, with one entity or one business unit, and then run into pricing structures that punish expansion. Commercial diligence here is ordinary software diligence with one addition, which is that the pricing metric often tracks the thing you most want to do more of.

‍

Find Out What the Price Scales With

‍

Ask which unit drives the price, whether that is entities quantified, revenue, assets under analysis, number of users, or quantification runs per year. Model the cost of the program two years out at the scope you expect to reach, not the pilot scope, and ask for that number in writing. Ask what a second business unit costs and what a third costs, since the curve matters more than the entry price. Vendors who publish pricing information make this part of the evaluation shorter.

‍

Check the Vendor's Stability Against the Length of the Commitment

‍

Quantification programs compound in value over years, with historical results becoming the baseline against which everything later gets measured, so a vendor who disappears takes the baseline with them. Ask how long the company has been operating, who funds it, how many customers run the platform in production today, and whether the modeling team is employed in-house. Ask for SOC 2 Type II and ISO 27001 reports under NDA, and read the exceptions section instead of the cover page.

‍

• Reference calls: Ask for two customers at your scale who implemented in the last year.
• Data exit: Ask what leaves with you if the contract ends, in what format.
• Roadmap commitments: Ask which promised features exist today and which are planned.

‍

Ask the Reference Customers What Went Wrong

‍

Vendor-supplied references are selected, and the useful question accounts for that selection. Ask the reference what surprised them during implementation, what they would scope differently, what the vendor handled well, and which stakeholder took the longest to convince. Ask whether the quantification results have ever been challenged internally and how that went. References answer these questions honestly far more often than buyers expect, because the person on the call remembers their own evaluation.

‍

Scoring the Answers Matters More Than Collecting Them

‍

Run the same questions past every vendor on the shortlist and write the answers down in the same structure, since comparison is the entire point and memory flatters whoever presented last. Weight the sections according to what the program has to accomplish. A public company facing disclosure obligations should weight model defensibility and reporting heavily. A buyer whose first job is winning a budget argument should weight the control-upgrade analysis and the board output. A buyer with a thin risk team should weight implementation effort and staffing above almost everything else, because a defensible model nobody has time to run produces nothing.

‍

Treat hedged answers as information instead of evasion. A vendor who says implementation takes four to nine months depending on asset data quality is being accurate, and a vendor who promises six weeks for every customer is describing a sales target. The evaluation is looking for a platform that produces numbers the organization can defend under questioning, which is a different test from finding the platform with the most features on the comparison sheet.

‍

Kovrr's team runs these evaluations with buyers regularly, including ones who go on to buy something else. Book a walkthrough of the platform and bring the hardest questions on this list.