How to Evaluate a Shadow AI Detection Tool in a Trial

A shadow AI detection trial produces a number within days. The tool reports how many AI applications it discovered, and the number is usually larger than anyone expected, which feels like proof that it works.

‍

It is not. Every tool in this category reports what it found. None of them reports what it missed, and the thing a buyer needs to know is the size of the second number.

‍

Why a Detection Trial Is Hard to Score

‍

Because the quantity being measured is invisible by definition.

‍

Discovering 180 AI applications tells you nothing on its own. It is a good result if the true figure is 190 and a poor one if the true figure is 400. Nothing in the tool's own output distinguishes those two situations, and a longer trial does not resolve it, since a tool blind to a category of usage stays blind to it for 90 days as readily as for 30.

‍

So the work sits before the trial rather than during it. Coverage can only be measured against something known independently.

‍

How to Establish What Should Be Found

‍

Three sources, none of which come from a detection tool.

‍

Seed Known Usage

‍

Before the trial starts, have a small group use a defined set of AI tools in defined ways. One person on a personal account rather than a corporate one. One using an AI feature inside software the company already licenses. One using a desktop application rather than a browser. One installing an AI browser extension.

‍

Each of those targets a specific blind spot, and the trial then has answers to check against. A tool reporting 180 applications while missing four known ones has told you something its own dashboard never would.

‍

Ask the People Using It

‍

An anonymous usage survey reaches the one place no technical control does, which is a personal device nobody administers. The results will be incomplete and will still be the only read available on that population.

‍

Read the Expense Records

‍

Individual subscriptions to AI services appear on expense claims and corporate cards. That list is independent of every detection method and frequently contains tools no technical control surfaced.

‍

Make the Methods Argue

‍

The disagreement between two sources is a measurement in itself.

‍

Run network or proxy data alongside browser-level telemetry for the same period, then compare. The overlap is what both methods see. What each one holds alone is a map of the other's blind spots, and the union is a floor on the true figure rather than the figure itself.

‍

Identity logs can join the same comparison. They see what authenticates through corporate identity, which excludes the free tier and the personal login entirely, so a tool resting on identity data alone will return a confident and small answer.

‍

Why the Union Keeps Growing

‍

If adding a third source keeps surfacing applications the first two missed, the estimate has not converged and the real population is larger than any single tool is reporting. That is a more useful trial finding than any discovery count.

‍

Five Tests Worth Running

‍

Each one targets a structural blind spot rather than a product weakness, so a tool failing them is not defective. It is seeing the slice its method can see.

‍

• Personal account on a sanctioned tool: the same application a company approves, accessed with a personal login. Identity and vendor-side reporting both miss it. Ask whether the tool distinguishes the account type rather than the application.
• AI inside licensed software: a feature switched on within a platform already in the estate. It generates no new vendor relationship, no new login and no unfamiliar domain.
• Usage outside the browser: a desktop application, an IDE assistant or a command line tool. Browser-level methods see none of it, which matters more as engineering teams adopt AI faster than other functions.
• An AI browser extension: installed by an employee, reading page content, which is a different artifact from visiting an AI site and is missed by methods watching destinations.
• An agent or connector: something granted access to a system and acting on its own, where the detection question is whether anything is recorded at all.

‍

A comparison of the tools in this category covers which products address which of these.

‍

What to Measure Besides the Count

‍

Four things, each of which decides whether a discovery is usable.

‍

Attribution. Whether a detection resolves to a named person or stops at a device or an IP address. An unattributed finding cannot start a conversation and cannot be governed.

‍

Account type. Whether corporate and personal use of the same tool are distinguishable, since the governance response differs completely between them.

‍

What moved. Whether the tool reports that sensitive data was submitted rather than that a site was visited. A list of domains is an inventory. A record of regulated data leaving is an exposure.

‍

Time to first detection. How long between an employee starting with something new and the tool reporting it. Weekly batch discovery and detection at the moment of use are different products wearing the same label.

‍

Questions to Ask the Vendor

‍

Four that are hard to answer with a demo.

‍

• What does your method structurally miss? A vendor who answers plainly is describing their architecture. One who claims complete coverage is describing their marketing.
• How do you detect something that is not in your catalog? Catalog matching finds known services. A tool new to the market last month is not in anyone's catalog.
• What does coverage depend on? An extension covers machines where it is installed. An agent covers managed endpoints. Unmanaged and personal devices sit outside both.
• What happens after discovery? Whether the finding becomes an inventory entry with an owner and a status, or a line in an alert feed nobody owns.

‍

What a Good Trial Produces

‍

Not a count, but a bounded estimate and a list of what the tool cannot see.

‍

That second list is what makes the first one usable. A security team knowing a tool covers browser-based usage on managed devices and nothing else can decide whether the remainder matters, add a second source, or accept the exposure deliberately. A team holding only a discovery count has a number it cannot interpret.

‍

The approach that holds up over time triangulates rather than choosing. Identity, network, browser and endpoint each contribute a partial record, and correlating them produces findings no single source supports. Shadow AI detection built on connected telemetry is the form that takes, with the browser layer supplying the part identity and network data cannot reach, including personal accounts on sanctioned tools.

‍

Scoring the Trial

‍

A detection tool reports what it found, and no tool reports what it missed, so a discovery count on its own cannot be read as good or bad. Establish ground truth first through seeded usage, an anonymous survey and expense records. Run two methods against each other and treat their disagreement as the measurement. Test the five cases that break each method, which are the personal account, the embedded feature, the non-browser application, the AI extension and the agent. Then measure attribution, account type, what data moved and time to first detection, because a finding nobody can act on is not coverage. The trial that ends with a list of what the tool cannot see is the one that was worth running.

‍

To see what connected telemetry surfaces across identity, network, browser and endpoint, book a demo with our AI risk experts.