AI Vendor Questions, and How to Check the Answers

Most security teams now run an AI-specific questionnaire alongside the standard one, often forty to sixty questions long, and the lists have converged. Training on customer data, retention, subprocessors, model provenance, human oversight, logging, incident response.

‍

Having the list is no longer the hard part. Every answer arrives as a sentence from the vendor, and a sentence is not evidence.

‍

Why the Standard Review Misses AI Vendors

‍

Because the questions it was built around assume software that does the same thing every time.

‍

Established questionnaires cover access control, encryption, segmentation and breach history well. They were not designed for a supplier whose product sends your content to a model somebody else operates, retains it for a period set by that third party, changes model versions without telling anyone, and depends on providers several layers removed from the contract you signed.

‍

AI questions have since been added to the major question sets and draw on NIST AI RMF, ISO/IEC 42001 and the OWASP list for LLM applications. The additions are sound. What has not caught up is verification.

‍

Which Answers Need More Than a Yes?

‍

Five, and each one has a version of the answer that is technically true and practically useless.

‍

Training on Customer Data

‍

This is not one question. Training, fine-tuning, evaluation, benchmarking, quality review, safety filtering and abuse monitoring are separate purposes, and a vendor can deny the first while doing several of the others. A denial that goes quiet once the remaining purposes are listed by name is the signal worth watching for.

‍

Two follow-ups do the work. Whether a no-training commitment covers customer-specific model variants that persist after a session, since a named fine-tune for one account can be argued as something other than general training. And whether an opt-out is contractual or a setting in an admin console that any administrator can switch back.

‍

Retention

‍

Ask what is retained rather than for how long. Prompts, uploaded files, outputs, embeddings and metadata have separate lifecycles, and embeddings in particular outlive the content they were derived from. Then ask whose clock governs, since a vendor retaining nothing while its model provider retains for thirty days has given a true answer and told you little.

‍

Model Provenance

‍

Which model served a given output, and whether the version is recorded alongside the record. Without that, a change in behavior six months from now cannot be attributed to anything, and a provider swapping a model underneath the vendor arrives with no notice at all.

‍

Subprocessors

‍

The weakest link in most reviews. Roughly two thirds of AI vendors do not disclose third-party subprocessors at all, and the common omission is the foundation model provider itself, kept outside the data processing agreement as though it were infrastructure rather than a processor.

‍

The checkable version asks for every provider in the request path by name, whether each appears in the DPA, whether obligations flow down, and whether plugins, connectors and marketplace extensions inherit the same guarantees. They usually do not.

‍

What Happens When the Vendor Ships a New Model

‍

The question most questionnaires skip, and the one that decides whether the whole review stays valid. A new model can change what data leaves, which provider receives it, how outputs behave and which regulatory tier the system falls into. Ask what triggers a notification, how much notice is given, and whether the answers to every question above are reconfirmed when it happens.

‍

How to Check Rather Than Accept

‍

Each claim has an artifact that would prove it, and asking for the artifact rather than the assertion is the whole discipline.

‍

• Training and retention: the relevant clause in the contract or DPA rather than a trust page, plus the model provider's own published terms as of the contract date.
• Subprocessors: the named list with a change-notification clause, checked for whether the foundation model provider appears on it.
• Isolation: how tenant separation works at the model and retrieval layer, since a vendor-held index over your documents is not the same as retrieval that stays local, whatever it is called.
• Deletion: how a deletion request propagates to subprocessors and model providers, and what confirms it completed.
• Framework alignment: the evidence behind a claim of alignment with ISO 42001 or NIST AI RMF, since alignment is a word anyone can use.

‍

The Failure None of These Questions Catches

‍

An ordinary one, and the record suggests it is the likelier event.

‍

In late 2024, Serviceaide, a vendor providing agentic AI-based IT management software, left an Elasticsearch database holding data for a client health system publicly accessible for roughly 47 days. The exposure affected 483,126 patients and included names, Social Security numbers, dates of birth and medical and insurance information. It was discovered in November 2024 and reported to federal regulators the following May.

‍

No model misbehaved. No agent exceeded its permissions. A database was misconfigured, which is the oldest failure in the catalog, at a company whose product happens to be agentic.

‍

Which Means the AI Questions Are Necessary and Not Sufficient

‍

A review that asks only about training, provenance and subprocessors would have cleared that vendor. The questions were pointed at the model layer while the exposure sat in infrastructure hygiene.

‍

So the AI questionnaire belongs alongside the conventional one rather than instead of it. An AI vendor is still a vendor with servers, databases, access controls and change management, and the thing that reaches your data may never touch a model at all.

‍

What Happens After Approval?

‍

The answers start expiring, which is the part a point-in-time review cannot address.

‍

A vendor assessed in March can change model providers in May, add a subprocessor in July and ship a new model in September. Nothing in the original questionnaire detects any of it, and the review's accuracy decays quietly from the day it is signed.

‍

Continuous third-party monitoring tracks model updates, capability expansions, new AI integrations, certification changes and incident history as they happen, and maps dependencies across vendors and their sub-vendors so the providers sitting behind your provider are visible rather than inferred.

‍

Checking One Vendor Before the Full Review

‍

Not every evaluation justifies a sixty-question process, and most teams want a read on a vendor before committing to one. The AI vendor risk catalog covers more than 15,000 providers and answers the narrower question of what is already known about one of them, which is enough to decide whether the longer review is worth running.

‍

The List Is the Easy Part

‍

Every AI questionnaire now asks about training, retention, provenance, subprocessors and model changes, and the lists have largely converged. What separates a review from a paperwork exercise is whether each answer arrives with the artifact that proves it, a contract clause rather than a trust page, a named subprocessor list that includes the foundation model provider, a deletion path that reaches every processor in the chain. The AI-specific questions also need the conventional ones beside them, since the largest exposure yet attributed to an agentic AI vendor came from a misconfigured database rather than from anything a model did. And because every answer starts expiring the day it is given, the review has to continue after the contract is signed. Kovrr assesses AI vendor risk continuously and quantifies what the exposure is worth, and a comparison of the tools in this category covers how the options differ.

‍

To check what is known about a vendor before the full review begins, book a demo with our AI risk experts.