AI Security and Governance Glossary

Adversarial attacks on AI are inputs deliberately crafted to manipulate a model's behavior, causing it to misclassify data, produce unsafe outputs, leak training information, or take actions the developer did not intend.

Agentic AI describes AI systems that plan, decide, and act autonomously across multiple steps and external tools, going beyond generating responses to executing actions on behalf of users or organizations.

Agent permissions and scoping define the specific tools, data sources, and actions an AI agent is authorized to access, applying least-privilege principles to autonomous AI systems the same way they are applied to human users.

AI accountability is the assignment of clear responsibility for the decisions, behavior, and outcomes of AI systems across their design, deployment, and operational lifecycle.

AI adversarial testing is the practice of deliberately probing AI systems with manipulated inputs, edge cases, and known attack techniques to evaluate their resilience against exploitation before and during production deployment.

An AI agent is a software entity that uses an AI model, typically an LLM, to plan, decide, and take actions in real systems, often chaining tool calls across multiple steps to accomplish an objective.

AI asset discovery is the process of identifying every AI system, model, agent, tool, and integration in use across an enterprise, including sanctioned deployments and unsanctioned shadow AI.

An AI asset inventory is the maintained catalog of every AI system, model, agent, tool, and integration in use across an enterprise, enriched with metadata on ownership, risk classification, data access, and business context.

AI assurance is the structured set of evidence, processes, and controls that demonstrate an AI system meets defined standards for safety, fairness, security, transparency, and regulatory compliance.

The AI attack surface is the total set of points where an AI system's models, training data, inputs, tools, integrations, and outputs could be exploited or manipulated by an adversary.

AI bias is systematic error in model outputs that produces unfair, inaccurate, or discriminatory results, often reflecting patterns present in training data, model design, or deployment context.

An AI Bill of Materials (AIBOM) is a structured inventory documenting every component of an AI system, including models, datasets, libraries, frameworks, and external dependencies, along with their sources and versions.

AI blast radius is the scope of potential impact if an AI system is compromised, defined by the data, tools, permissions, and downstream systems that a single AI system, agent, or model can reach.

AI compliance is the practice of meeting the regulatory, contractual, and internal-policy obligations that govern how AI systems are developed, deployed, and operated across an enterprise.

AI data governance is the framework for managing data quality, provenance, privacy, and access across the AI lifecycle, covering training data, fine-tuning data, retrieval sources, and inputs and outputs at runtime.

AI data leakage is the unintended exposure of sensitive information through an AI system, whether via user inputs sent to external models, model outputs that reveal training data, or agent actions that expose data through connected tools.

AI explainability is the ability to describe how an AI system arrived at a specific output or decision, in terms a human reviewer, auditor, or affected party can understand and evaluate.

AI exposure is the quantified financial impact an organization faces from AI-related risks, spanning security incidents, governance failures, regulatory penalties, and downstream business consequences of AI system behavior.

AI fairness is the principle that AI systems should produce outcomes that do not unjustly discriminate against individuals or groups, particularly across protected characteristics like race, gender, age, and disability.

AI governance is the framework of policies, controls, roles, and accountability structures that direct how an organization develops, deploys, monitors, and manages AI systems across their entire lifecycle.

An AI governance program is the operationalized set of people, processes, and technology that carries out an organization's AI governance policies day-to-day, translating policy into consistent execution across every AI system in the enterprise.

AI guardrails are runtime controls that constrain an AI system's behavior during inference, blocking unsafe outputs, restricted actions, or policy violations before they reach users or downstream systems.

An AI impact assessment is a structured evaluation of the potential effects an AI system may have on individuals, groups, business operations, and regulatory obligations, conducted before or during deployment to identify and mitigate risks.

AI incident response is the structured process of detecting, containing, investigating, and recovering from failures, misuse, or security events involving AI systems, extending traditional incident response into AI-specific failure modes.

An AI loss scenario is a defined event involving AI systems, modeled with probability and financial impact distributions, used within an AI risk quantification program to produce enterprise exposure figures and prioritize mitigation.

AI policy is the set of internal rules that define how an organization approves, deploys, and manages AI systems, what AI use is permitted, what is prohibited, and who is responsible for enforcement.

AI red teaming is the practice of using adversarial testers, often working creatively rather than systematically, to probe AI systems for safety, security, and misuse failures that structured testing does not surface.

AI risk management is the discipline of identifying, assessing, mitigating, and monitoring risks specific to AI systems across their entire lifecycle, sitting as a function within the broader AI governance program.

AI risk prioritization is the process of ranking AI-related risks by likelihood, financial impact, and business exposure, so mitigation resources are allocated to the risks that materially affect the organization rather than distributed evenly.

AI Risk Quantification (AIRQ) is the practice of measuring AI-related risk in financial terms, translating security, governance, and compliance failures into probabilistic dollar exposure that boards, CFOs, and risk teams can act on.

An AI risk register is the maintained catalog of AI-specific risks an organization faces, documenting each risk's scenarios, controls, ownership, quantified exposure, and current mitigation status.

AI security is the practice of protecting AI systems from attacks, misuse, and unauthorized access, extending traditional cybersecurity into the AI-specific threat surface across models, data, prompts, tools, and agents.

AI sprawl is the uncontrolled proliferation of AI systems, tools, agents, and use cases across an enterprise, typically driven by decentralized adoption that outpaces governance and security oversight.

An AI supply chain attack compromises an AI system through its upstream components, including foundation models, datasets, open-source libraries, MCP servers, or third-party providers, rather than through the deployed system directly.

AI supply chain risk is the exposure an organization inherits from every AI component it does not directly control, spanning foundation models, training datasets, open-source libraries, MCP servers, and third-party AI providers.

AI third-party risk management is the discipline of assessing, contracting for, and continuously monitoring the risks introduced by AI vendors, model providers, and connected AI services that touch enterprise data and systems.

AI transparency is the practice of disclosing meaningful information about AI systems, including their intended use, training data, capabilities and limitations, and the logic behind their outputs and decisions.

An AI use case inventory is the catalog of how AI is being applied across an enterprise, mapping specific business purposes and workflows to the AI systems supporting them, along with associated risk classifications and controls.

AI vendor risk is the exposure an organization faces from third-party AI providers, spanning foundation model vendors, AI-enabled SaaS, and any external service where AI processes enterprise data or drives enterprise decisions.

Artificial Intelligence (AI) refers to computer systems designed to perform tasks that typically require human intelligence, including perception, reasoning, learning, decision-making, and language understanding.

An autonomous AI system operates and makes decisions without direct human intervention at each step, executing multi-step tasks based on defined objectives, learned behaviors, and access to tools or actions.

The Colorado AI Act (SB 205), signed in May 2024, is the first comprehensive AI law in the United States, regulating developers and deployers of high-risk AI systems that make consequential decisions affecting individuals in Colorado.

Continuous AI monitoring is the ongoing observation of AI system inputs, outputs, behavior, and performance in production, detecting model drift, misuse, security events, and control failures as they occur.

Data poisoning is an attack in which an adversary corrupts training, fine-tuning, or retrieval data to cause an AI model to behave in specific unwanted ways, embed backdoors, or produce biased or unsafe outputs.

Enterprise AI adoption is the process by which organizations move AI from initial experimentation into governed, scaled deployment across business operations, balancing innovation with risk management, security, and compliance.

The EU AI Act (Regulation (EU) 2024/1689) is the European Union's comprehensive AI regulation, imposing risk-based obligations on providers and deployers of AI systems placed on the EU market or used in the EU.

Article 50 of the EU AI Act imposes transparency obligations on providers and deployers of certain AI systems, requiring disclosure of AI-generated or manipulated content, notification of AI interactions, and machine-readable marking of synthetic content.

A high-risk AI system under the EU AI Act is an AI system meeting the criteria in Article 6 and Annex III, subject to conformity assessment, risk management, technical documentation, transparency, human oversight, and post-market monitoring obligations.

EU AI Act prohibited AI practices are AI use cases banned outright under Article 5 of the Regulation, including manipulative or exploitative AI, social scoring by public authorities, and untargeted biometric database scraping.

A foundation model is a large AI model trained on broad, general data at scale, designed to be adapted to a wide range of downstream tasks through fine-tuning or prompting, forming the base layer of most current enterprise AI systems.

General-Purpose AI (GPAI) refers to AI models trained on broad data at scale and capable of performing a wide range of distinct tasks, a category defined and regulated specifically under the EU AI Act.

Generative AI refers to AI systems that produce new content such as text, images, audio, video, or code by learning statistical patterns from training data and generating novel outputs in response to prompts.

A GPAI model with systemic risk is a general-purpose AI model meeting EU AI Act criteria for high-impact capabilities, subject to additional obligations including model evaluation, adversarial testing, incident reporting, and cybersecurity requirements.

Hallucination is a failure mode of generative AI in which a model produces content that is plausible-sounding but factually incorrect, fabricated, or unsupported by any real source, presented with the same confidence as accurate output.

Human-in-the-loop (HITL) is an AI oversight approach in which a human reviews, approves, or intervenes in AI decisions and actions at defined checkpoints before those decisions or actions take effect.

Human-on-the-loop (HOTL) is an AI oversight approach in which a human monitors an AI system's operation continuously or periodically and can intervene when necessary, without being required to approve each individual action.

Indirect prompt injection is an attack in which malicious instructions are hidden in content that an AI system later processes—web pages, documents, emails, tool outputs—hijacking the model's behavior without direct interaction with the attacker.

ISO/IEC 23894 is the international standard providing guidance on AI risk management, extending the general risk management guidance of ISO 31000 with AI-specific considerations across the AI lifecycle.

ISO/IEC 42001 is the international standard specifying requirements for an AI management system, providing a certifiable framework for how organizations govern the development, provision, and use of AI systems.

An AI jailbreak is an input crafted specifically to bypass a model's safety controls, guardrails, or usage policies, causing the model to produce content or take actions the developer had blocked.

A Large Language Model (LLM) is a type of AI model trained on massive amounts of text data to understand and generate human language, forming the technical foundation of most enterprise generative AI and agentic AI systems.

Machine Learning (ML) is a subset of AI in which systems learn patterns from data and improve their performance on tasks without being explicitly programmed with rules for every case.

MCP security is the practice of securing Model Context Protocol implementations, addressing tool poisoning, over-broad permission scopes, credential handling, and supply chain risk in the MCP ecosystem.

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a knowledge base of adversarial tactics and techniques targeting AI systems, providing a shared taxonomy for AI security testing, threat modeling, and defense.

A model card is a structured document describing an AI model's intended purpose, capabilities, limitations, evaluation results, training data characteristics, and appropriate use cases, produced by the model developer for downstream users.

Model Context Protocol (MCP) is an open standard developed by Anthropic that defines how AI systems connect to external tools, data sources, and services, enabling agents to access enterprise systems through a consistent interface.

Model drift is the degradation of AI model performance over time, occurring when real-world data, user behavior, or usage patterns diverge from the distribution the model was trained on.

Model extraction is an attack in which an adversary reconstructs an AI model's parameters, decision boundaries, or capabilities by querying it strategically, enabling model theft or preparation of downstream attacks.

Model poisoning is an attack that manipulates a model's weights, fine-tuning process, or update mechanism to embed backdoors, biases, or malicious behaviors directly into the trained model itself.

A multi-agent system is a group of AI agents that coordinate, communicate, and delegate tasks among themselves to accomplish objectives that would exceed the capacity of any single agent.

The NIST AI Risk Management Framework (AI RMF 1.0) is a voluntary framework published by the US National Institute of Standards and Technology for managing risks associated with AI systems across the full lifecycle.

The Govern function of the NIST AI RMF establishes the organizational culture, policies, processes, and accountability structures that underpin AI risk management, applying continuously across the Map, Measure, and Manage functions.

Personally Identifiable Information (PII) in AI refers to personal data that enters or exits AI systems through prompts, outputs, training data, retrieval sources, or agent-executed actions, creating specific privacy exposure paths.

Prompt injection is an attack in which crafted input causes an AI model to follow attacker-controlled instructions instead of, or in addition to, the developer's intended instructions.

Responsible AI is the practice of developing, deploying, and operating AI systems in ways that are fair, transparent, accountable, secure, and aligned with ethical, legal, and societal norms.

Retrieval-Augmented Generation (RAG) is an architecture that combines an LLM with a retrieval system, grounding the model's outputs in documents fetched from an external source at inference time.

Sensitive data categories in AI include personal, financial, health, biometric, and other regulated data types that require heightened protection when processed by, or exposed to, AI systems.

Shadow AI is the unsanctioned use of AI tools, models, and systems across an enterprise, adopted by individual employees or teams without formal governance, IT, or security approval.

Tool poisoning is an attack in which malicious instructions are embedded in the tool descriptions or metadata that an AI agent reads when selecting or using tools, hijacking the agent's behavior through the tool interface itself.

AI training data is the dataset used to teach a machine learning model, encompassing raw examples, labels, and any data used during initial training or subsequent fine-tuning that shapes the model's learned behavior.