The Three Questions Every AI Telemetry Claim Should Survive

A Diagnostic Test, and Five Ways Agent Telemetry Fails It

‍

Coding-agent telemetry, today, cheaply, answers four real questions: which agents are running and operated by whom, what an agent invoked, what happened in a session in order, and whether a run looks abnormal. Part 1 of this series covers that case in full.

‍

This part is about the fifth question every security team eventually asks, the one no amount of instrumentation answers on its own: can this record be trusted enough to build a control on it? Before any telemetry source, an AI agent's or anything else's, gets to carry an assurance claim, run it through three questions.

‍

Three Questions to Ask Before Any Telemetry Carries an Assurance Claim

‍

This is the reusable part, and it applies well beyond AI agents.

‍

1. Who controls the emitter, relative to who is being observed?
2. Is the record complete, or sampled?
3. Does it record the attempt, or the outcome?

‍

Agent telemetry answers: the subject controls it, sampling is architecturally available, and it records the attempt. Three out of three on the unhelpful side. Instrument anyway, and know precisely what the record is before building a control on top of it.

‍

Run the same three questions against the sources already in a SOC, and the contrast is sharp. AWS CloudTrail is written by the control plane rather than by the caller, retains every management event rather than a sample, and records the outcome including the authorization decision. Three out of three the other way. The completeness has a seam even here: the object-level data events closest to what an agent's tool call actually touches, which includes an S3 read or a table scan, are opt-in and billed per event, not on by default the way management events are. That is why an investigator reaches for it first, and it is the standard agent telemetry has to be measured against rather than graded on its own curve.

‍

Each of the three deserves its own treatment, because the mitigations differ.

‍

The Emitter Is the Subject

‍

The agent writing the telemetry is the agent under review. Every one of the five agents tested in the field test behind this series reads its telemetry configuration from a file or an environment variable in the developer's own home directory: $CODEX_HOME/config.toml for Codex, .gemini/settings.json for Gemini, OTEL_EXPORTER_OTLP_ENDPOINT for Claude Code and Copilot. An agent run with any of those switched off produces nothing, and nothing looks exactly like a quiet week. There is no negative signal, so a serious deployment needs alerting on stream cessation and on writes to telemetry configuration, and almost nobody ships that.

‍

A competent reader will object that this describes every log, and the objection deserves a real answer rather than a dismissal. It is not true of the sources security teams already lean on hardest. Cloud audit logs are written by the cloud provider's control plane, not by the caller. Identity logs are written by the identity provider, not by the principal. Those records are produced by something structurally separate from the actor they describe, which is precisely why they carry weight in an investigation.

‍

Agent telemetry has no such separation. The distinction worth drawing is narrower than self-reported versus authoritative: it is whether the party with a motive to omit something is also the party writing the record.

‍

This is not unique to OpenTelemetry, or to this series' own testing. An independent teardown of GitHub Copilot's separate control-plane audit log, a different product from anything OTel-based, found the same gap: audit records carry no full local IDE or CLI session, no prompts, no responses, no generated code, no local tool calls. Exporting OTel data alongside it closes part of that hole, but prompt and tool-argument content stays excluded by default there too, requiring an explicit opt-in. Two unrelated signals from the same vendor, an audit log and a telemetry pipeline, share the same blind spot for the same reason.

‍

There is one partial answer, and it is worth knowing about. Both Claude Code and GitHub Copilot now support administrator-imposed telemetry configuration delivered by MDM or account policy, which GitHub's July 2026 changelog describes as taking "precedence over environment variables and user settings". That moves the switch out of the observed party's hands, which is a real improvement over a developer-owned config file. It does not make the agent's account of its own behavior independent of the agent.

‍

It Records Intent, Not Effect

‍

This is the limitation that survives every improvement in instrumentation, and it is the one most often missed.

‍

A tool span records that an agent invoked something. In the field test, Codex emitted a codex.tool_result event carrying both the shell command it ran and the command's full standard output, which is about as complete as an intent record gets. It still does not record whether the credential had permission, whether the resource existed, or whether anything changed, because the agent's own process is not where that is decided. Swap the harmless fixture command for one that deletes an object from a production bucket and the span looks identical either way: same shape, same attributes, same success flag from the shell's point of view.

‍

That swap already happened once, for real. Orca Security's February 2026 disclosure of RoguePilot showed a hidden instruction inside a GitHub Issue, invisible to a human reviewer in an HTML comment, directing GitHub Copilot to check out a crafted pull request and exfiltrate the workspace's GITHUB_TOKEN through a symlinked file. Copilot's own span for that turn records a tool call and a completed status, the same shape as checking out any other branch. The effect, a stolen token with full repository write access, lived only in what happened next.

‍

Agent telemetry is an intent record. Cloud audit logs, git history, database audit trails and identity logs are effect records. Attribution requires both, joined. A perfectly conformant, fully complete, tamper-proof agent telemetry stream would still only tell you what the agent tried.

‍

The three questions above are about whether to trust a record at all. A separate cluster of problems sits alongside them, less about architecture and more about practice: whether the vocabulary two vendors use actually means the same thing, whether that vocabulary is stable enough to build detection content against, whether it has a word for the one signal compliance asks for by name, and whether the vendor's own documentation about any of this can be taken at face value. All four showed up in the same field test.

‍

Comparability Across Vendors Is Not Given

‍

The premise of a shared vocabulary is that a query written once works everywhere. For coding agents in 2026, it does not.

‍

Measured across five agents on an identical task, the share of span attributes sitting in the standard gen_ai.* namespace ranged from 5% to 100%. One agent reports token counts as gen_ai.usage.input_tokens and another as plain input_tokens, so a rule written against the convention silently returns nothing for the second. Detection content written against the standard covers whichever subset of an estate happens to conform this quarter, and produces no error for the rest.

‍

Nothing In the GenAI Conventions Is Stable Yet

‍

OpenTelemetry itself graduated from CNCF in May 2026. The GenAI conventions have not.

‍

OpenTelemetry's own blog says so directly. Its most recent post on AI agent observability describes agent-framework conventions as still being defined through open discussion, only a draft agent-application convention as "finalized," and the conventions for models themselves as "experimental." That is not an outside read of the project's maturity. It is the project's own.

‍

At the commit examined for this research, the GenAI semantic conventions repository carried 614 commits, 80 of them in the preceding 60 days, and zero tagged releases. Across its model definitions, there are 188 stability declarations, and every single one reads development. The only attributes marked stable that appear on a GenAI span are general-purpose ones borrowed from the core conventions: error.type, server.address and server.port.

‍

A detection rule keyed on gen_ai.tool.name therefore rests on a name the project has explicitly declined to promise. When a name changes, nothing raises an error. The rule stops matching, quietly, and the dashboard stays green.

‍

The Vocabulary Has No Word For a Human Decision

‍

Regulators have already decided that a human decision is a minimum logged fact. Article 12(3) of the EU AI Act, setting out minimum logging for biometric identification systems, names "the identification of the natural persons involved in the verification of the results" among the things a log must record.

‍

That clause governs biometric systems rather than coding agents, and the principle it encodes is not category-specific. When an automated system acts, who approved it is part of the record. Checking every one of the 63 gen_ai.* and 4 mcp.* registry attributes at the examined commit returns no attribute for approval, decision, consent, permission, or any human-in-the-loop concept.

‍

Four of the five agents tested emit one regardless, in their own namespaces. Gemini CLI goes furthest and distinguishes an approval a person gave from one granted automatically. GitHub Copilot CLI, the agent that conforms most closely to the conventions, emits nothing of the kind.

‍

An organization that standardizes strictly on convention attributes today would discard the one signal a compliance function asks for by name.

‍

Documentation About Telemetry Safety Is Not a Control

‍

Two vendor claims about telemetry behavior were tested directly against a collector for this research. One was that GitHub Copilot CLI disabled export rather than sending data over plain HTTP; it exported 407 KB including prompt content, and logged no warning. The other was that Claude Cowork included prompt content by default; it did on one client surface and redacted it on another, under one administrator configuration, with no setting exposed for either.

‍

Both are documented behaviors that did not hold. The general lesson is worth more than either instance: telemetry configuration is increasingly pushed fleet-wide from an admin console, and the safety properties of those pipelines are being taken from vendor documentation rather than from measurement of one's own environment.

‍

None of the five sections above argue against instrumenting. They argue against stopping there. A self-reporting emitter, an intent-only record, an inconsistent vocabulary, an unstable convention, a missing approval field, a vendor claim that does not hold under measurement: every one of these is fixable, not by more OpenTelemetry on its own, but by something alongside it. Part 3 covers what that something is, what it costs to build, and what it still cannot promise.

‍

If an agent's own telemetry is a self-reported record of intent, the fix isn't more of it. It's a second source the agent doesn't write. See how Kovrr corroborates the agent's account against identity, cloud, and endpoint records it can't edit. Book a demo today. 

‍

About This Research 

‍

This series draws on a Kovrr Research field test of five production coding agents (Claude Code, OpenAI Codex CLI, Google Gemini CLI, GitHub Copilot CLI, and Claude Cowork), each instrumented and captured against a live OpenTelemetry collector on an identical task.

‍

Sources

• Field test of five coding agents (Kovrr Research)
• Claude Code: monitoring usage / OpenTelemetry
• Claude Code: environment variables reference
• Claude Code: managed settings reference
• "Enterprise-managed OpenTelemetry export for VS Code and CLI" (GitHub Changelog, 8 July 2026)
• GitHub Copilot CLI changelog
• GitHub Copilot: OpenTelemetry observability docs
• Codex CLI configuration reference (OpenAI)
• Gemini CLI telemetry documentation (Google)
• AWS CloudTrail user guide
• "GitHub Copilot Audit Logs" (Monad, July 2026)
• "RoguePilot: Critical GitHub Copilot Vulnerability" (Orca Security, February 2026)
• "AI Agent Observability" (OpenTelemetry Blog, 2025)
• open-telemetry/semantic-conventions-genai (GitHub repository)
• Regulation (EU) 2024/1689, Article 12: Record-Keeping (Article 12(3))