[Paper] Tracekit: Tamper-Evident Intent-Reasoning-Action Auditing for AI Agents

Bravish Ghosh, Cygnux Labs.

Download the PDF · arXiv:2609.35659 · Code (MIT) · Licence: CC BY 4.0

Abstract

AI agents run shell commands, edit repositories, send email and move money with their operators’ credentials, and the record of what they did is usually a log that the agent, or whoever runs it, can edit. We present Tracekit, an open-source system (MIT licence, version 1.4.1) that turns each agent run into evidence a third party can check offline. A signer that, outside dev mode, the agent cannot reach decides every tool call it is sent against its own policy before it runs (allow, deny, or hold for a person), and signs the call, the decision and the result as Ed25519 records over RFC 8785 canonical JSON, in a per-run hash chain inside an RFC 6962 Merkle tree. It checkpoints the tree as C2SP signed notes that independent witnesses can cosign. A run exports as one self-contained .tkb bundle that two independent verifiers (Python, and TypeScript, which also runs as a single browser page) check against a trust configuration the verifier pins, reporting integrity and assurance separately. Because a signer can sign only what reaches it, Tracekit captures activity at several layers (agent state, tool calls, model calls through an out-of-process gateway, network and kernel effects) and writes the disagreements it detects between layers as signed records, never as silence. On top of the evidence it adds two analyses: checks that compare what a person asked, what the model said and what ran; and causal attribution, which replays a run without each suspect input to find the one that caused an action.

This version replaces our earlier description of an unsigned, hook-written hash-chained ledger. We report 21 experiments and keep the misses. A signed log alone detects 6 of 8 tamper classes in every trial and a witness catches the other two; every case of two insider-attack suites (8 and 6 cases) against a Linux system-mode signer is caught, as a merge gate on every pull request; a tool call costs 0.45 ms at the median and 0.91 ms at p99 for the signer round trip (acknowledged on write); no acknowledged record was lost across 60 rounds of kill -9 under concurrent clients; cross-layer reconciliation flags 12 of 12 seeded discrepancies with no false positive; and the verifiers agree on all 63 bundles, rejecting all 33 forgeries. Policy rules are a tripwire, not the boundary: they stop 570 of 570 harmful calls in our corpus (one false alarm in 738) but 211 of 420 (50.2%) written blind to them; the protection is a signer and credentials the agent cannot reach. On a held-out phrase pool the intent–reasoning–action checks find a failed test run claimed as passing in 86% of synthetic sessions with 5 false findings and one false hold in 1,812 honest calls, and with a local 7B model, replay names exactly the planted cause in 6 of 6 harmful runs where that cause is confirmed, with no false blame. All code, data, results and the scripts that regenerate every table are public.

Figure 1: The architecture of Tracekit 1.4, by who can reach what. Outside dev mode the agent reaches the signer only through an RPC, and in every mode its process holds no key: it cannot assign sequence numbers, write gap records or claim its own isolation. Its hooks hand the signer the person’s prompt, and a transcript tailer, framework integration or the gateway what the model said, so a limit the request states is held before the call runs and an analyzer later compares what was asked, said and done. Services outside the agent hold the credentials and observe its effects. Witnesses the operator does not control can keep the operator honest. Anyone verifies a bundle offline against keys they pin, and gets integrity, assurance and signed findings as three separate results. † Experimental in 1.4; no public witness runs yet.

1 Introduction

AI agents are no longer chat windows. A coding agent such as Claude Code, Codex or Cursor reads a repository, runs shell commands, edits files, pushes branches and spawns sub-agents that work in parallel. An agent built on the OpenAI Agents SDK, LangGraph, CrewAI or the Model Context Protocol (MCP) issues refunds, sends email, queries production databases and calls cloud APIs. Each step runs with the permissions of whoever deployed the agent, and much of the text the agent reads (a README, a ticket, a web page, a tool result) can be written by someone else, which is the classic setting of indirect prompt injection [17, 45, 13].

When such an agent does something it should not have done, three questions follow:

Q1. What did it do? Every action, by every agent and sub-agent, in order, with its arguments, the rule that decided it before it ran, and who approved it.

Q2. Was it asked to? The request the action served, and any limit the person stated (“don’t touch billing.py”).

Q3. Did it say what it was doing? What the model told the person before and after the call, next to what actually ran.

Answering them after the fact requires a record that the parties involved could not have changed, that says what it failed to capture, and that ties each action to the request it served and to the agent’s own account of it. The records that exist today meet none of these needs. Coding agents keep transcripts in the user’s home directory, where the agent itself can rewrite them; a prompt-injected agent that can run shell commands can delete its own transcript. Frameworks and tracing platforms [35, 26] store traces in a database run by the operator, the party a customer would dispute with. Neither kind of record says which actions it missed, and neither binds an action to the request or to what the model said at the time. A trace is evidence only for someone who already trusts whoever stored it, and an agent steered by injected content, or pursuing a goal its operator did not intend [16, 25], is the one party whose account should never be the only one.

Approach. Tracekit builds such a record from three ideas (fig. 1).

1. The agent asks; a signer writes. Every tool call goes to a signer that, except in dev mode, runs outside the agent’s reach. The signer decides the call under its own policy, assigns its position, signs it and appends it to a log. The agent’s process never holds a key, and nothing it claims about itself (its isolation, its fail mode, a gap in its reporting, who approved a call) is taken on trust.

2. The operator is a separate adversary. Whoever runs the signer holds its keys and could rewrite its log. Tracekit therefore applies the design of transparency logs [28, 29] to agent actions: the log is a Merkle tree, its checkpoints follow the C2SP formats [5, 6], and independent witnesses can cosign them [7] (experimental; no public witness runs yet), so a rewrite of witnessed history is detected. A verifier reports integrity (is the record intact?) separately from assurance (who vouches for it?).

3. Audit intent, reasoning and action together. The signer commits to each prompt the person submitted (the stated intent) and to each message the model wrote before a call (its stated reasoning), next to the action itself. A limit stated in the prompt that the hook’s grammar can read becomes a live constraint, held by the signer before the next call runs. After the run, an analyzer compares the three, and the signer signs each finding into a findings run under the analyzer’s measured identity, reported beside the integrity verdict, never inside it.

A signer can sign only what reaches it, so the hard problem moves from integrity to completeness. Tracekit captures the same activity at several layers (saved state, tool calls, the model’s requests, network connections), compares them, and writes each disagreement it detects as a signed record; kernel events, where a sensor covers the run, are recorded and checked against the calls after the run. Silence becomes evidence. And because a model’s account of why it acted is testimony [43, 10], tracekit why answers which input caused this action by replaying the run with each suspect input removed and testing the change statistically.

The result is one offline-verifiable .tkb bundle per run that answers the three questions with evidence a third party can check, and says plainly what it cannot answer.

The claim this paper defends

Tracekit proves what its capture path recorded, and that the record has not changed since it was signed and checkpointed. It does not prove intent, that every action was captured, that a tool result the agent reported was true, or that what the model said was its real reasoning. “Intent” and “reasoning” in this paper always mean the stated request and the model’s stated account, recorded as such. A finding is a signed opinion about the record, never part of its integrity. Against the operator, the claim holds only for what an independent witness has seen. The claim is detection, not prevention.

Contributions.

1. A signer architecture for agent evidence (sections 3 and 4) with a named threat model of six adversaries, eleven invariants mapped to tests, and three separate guarantees (evidence integrity, execution control, independent assurance), each with what it needs.

2. Evidence format v2 (section 4.2): canonical JSON, domain-separated Ed25519 signatures, positional key validity without clocks, RFC 6962 trees, C2SP checkpoints and witness cosignatures, per-tenant registry logs that prove which runs exist, and a verifier that reports integrity and assurance separately. The format has normative test vectors, and two independent verifiers (one also built as a browser page) agree on all 63 bundles of E22.

3. Capture-layer reconciliation (section 4.5): an out-of-process LLM gateway, tool executor, MCP gateway and egress observer (all experimental), and signed records for the disagreements between layers.

4. Intent–reasoning–action auditing on signed evidence (section 5): the request and the model’s account committed by the signer and labelled by trust tier, live constraints that can only tighten a decision, checks of asked against said against done, and findings signed under a named analyzer’s identity and cited by record.

5. Verifiable causal attribution (section 6): replay with each suspect input removed, with paired, sequential and multiplicity-corrected tests.

6. An evaluation of 21 experiments (section 7), 17 of which run on every pull request, with the misses kept: what the log alone cannot see, a policy gate that stops half of a pool written blind to it, text checks that do not generalise to unseen wording, and a reviewer-model arm that has not been run yet.

What changed since version 1. Table 1 summarises the difference. Version 1 [15] described a dependency-free ledger written by a hook process running as the agent’s own user, hash-chained but unsigned, with external anchors as the only defence against a full rewrite. It already asked the three questions above, under the name intent–reasoning–action auditing. Its design had two weaknesses: the ledger was written as the agent’s own user, so the agent could rewrite it; and the model’s reasoning was read from a transcript the agent could also write. Its own evaluation showed the limits: tail truncation and re-chaining were invisible without anchors, the regular-expression gate blocked 18 of 44 harmful calls, and the provider withheld every thinking block. Version 2 rebuilds the record around a signer the agent cannot reach outside dev mode, and in release 1.4 intent and reasoning come back on that footing: committed by the signer, labelled by how far the agent could have written them, enforced where they state a limit, and compared with the action by an analyzer whose findings are themselves signed evidence.

Table 1: From version 1 (September 2026) to version 2 (this paper, Tracekit 1.4.1).

Roadmap. Section 2 places Tracekit among tamper-evident logs, transparency systems, agent observability and AI control. Section 3 gives the threat model, the guarantees and the deployments. Section 4 describes the signer, the evidence format, the verifier, the policy engine, capture and reconciliation, containment, the integrations and the viewer. Section 5 covers intent, reasoning and action, and section 6 causal attribution. Section 7 reports the experiments and section 8 a case study on two open-source agents. Section 9 discusses lessons, limits and future work. The appendices give the reproduction commands, sample verifier output, the finding catalogue and a glossary.

2 Background and related work

Table 2 positions Tracekit against the families of systems it borrows from or sits beside. We describe each family below.

Table 2: What each family of systems provides for agent evidence. ✓ provided, ∘ partial or optional, – not provided. A row describes the family as typically deployed, not every product in it.

Tamper-evident logging. Haber and Stornetta linked timestamped documents into a hash chain so that back-dating requires collusion [18]. Schneier and Kelsey protected audit logs on an untrusted machine with evolving MAC keys, so that an attacker who compromises the machine cannot read, or undetectably alter or delete, entries written before the compromise [40]; Ma and Tsudik gave forward-secure sequential aggregate signatures for the same setting with public verification [30]. Crosby and Wallach introduced history trees with logarithmic membership and consistency proofs [11], the basis of what transparency logs call consistency proofs. Tracekit’s per-run chains, positional key retirement and witnessed tree heads follow this line. The difference is the setting: in these schemes the logging host may be compromised later, while in Tracekit the party whose actions are logged (the agent) is never given a key outside dev mode, and the logging party (the operator) is modelled as a separate adversary.

Transparency logs, Sigstore and C2SP. Certificate Transparency publishes certificates in append-only Merkle trees whose signed tree heads are checked by monitors and auditors, so a mis-issued certificate cannot be hidden [28, 29]. Sigstore makes software signing usable with short-lived certificates bound to an identity (logged in a certificate-transparency log), with signatures recorded in the Rekor transparency log [34, 41]. The C2SP specifications standardise signed notes, checkpoints, tiles and the witness protocol, so that independent witnesses can cosign any log’s checkpoint after checking its consistency [5, 6, 8, 7, 9]. Tracekit follows those specifications exactly, so standard witnesses can cosign Tracekit logs without knowing anything about agents, and it can anchor checkpoints in Rekor v2 with an RFC 3161 timestamp [1] (experimental; Sigstore staging only until production lists Rekor v2). Its record-key issuer follows Sigstore’s pattern of short-lived keys whose certificates are logged before use. What Tracekit adds is the content and the protocol around it: what a record of an agent action must contain, who may write it, and how a missing record becomes visible.

Agent observability. Tracing tools for LLM applications, including LangSmith, Langfuse, Arize Phoenix and exporters that follow the OpenTelemetry semantic conventions for generative AI [35], record prompts, model calls and tool calls for debugging and evaluation. Claude Code exports OpenTelemetry events and exposes lifecycle hooks [2], and community plugins forward its sessions to Langfuse [26]. These are the right tools for debugging. Their records are not, as far as we know, signed or independently witnessed. Tracekit imports OTLP traces as tier T3 (proof of receipt, nothing more) and exports its own records as OpenTelemetry spans that carry their entry hashes, so it complements these tools rather than replacing them.

Agent evidence layers. Closest to Tracekit are systems that pair agent execution with a verifiable record. nono combines OS-level sandboxing with a Merkle-tree audit log whose root the supervisor outside the sandbox can sign [19]. halo-record provides a hash-chained evidence layer with an optional third-party witness and adapters for several agent frameworks [24]. Tracekit differs in three ways that our evaluation exercises: the decision is made by a policy engine in the signer over each tool call’s arguments before it runs, not by OS-level sandbox rules, every capture layer’s report is reconciled against the others with signed gap records, and the verifier separates integrity from assurance so that a run no witness has seen can never be reported as more than dev, and one only the operator witnessed as no more than local.

Guardrails, sandboxes and capability systems. NeMo Guardrails [37], Llama Guard [20] and Invariant [21] decide whether a model input, output or action is acceptable. CaMeL separates control flow from untrusted data so that injected text cannot choose actions [12]. ToolEmu uses an LM-emulated sandbox to surface risky agent actions [38], and R-Judge benchmarks how well LLMs judge safety risks in agent interaction records [44]. These systems decide or contain; Tracekit’s signer decides too, with its own packs, but its contribution is the evidence that a decision was made, what it saw, and who approved it. A sandbox’s own decisions (NVIDIA OpenShell’s blocks, a kernel sensor’s observations) can be signed into the same run (section 4.6).

AI control and chain-of-thought monitoring. AI control asks how to use a model safely even if it is trying to subvert its overseers, with trusted monitors, resampling and editing of suspicious actions [16, 4]; SHADE-Arena measures whether agents can sabotage while evading a monitor [25]. A monitor’s input is the transcript, and a transcript the monitored agent can edit is a weak input. Chain-of-thought monitoring is useful but fragile [23, 3], and stated reasoning is often unfaithful [43, 27, 10]. Tracekit treats what the model says as testimony and what ran as evidence, signs both, and answers “why” with interventions on the inputs (section 6) rather than with the model’s own account. Its reviewer-model finding (TK-X210) is a monitor pattern from the control literature (experimental, not yet evaluated) whose verdict is signed and marked as an opinion, never part of integrity.

Counterfactual attribution. Removing an input and observing the change in an outcome is the interventional notion of cause [36]. With stochastic agents the change must be estimated from repeated runs; Tracekit pairs runs on common seeds and replays recorded tool results from a tape so that only the intervention differs, and uses exact McNemar tests [31], Newcombe’s hybrid score interval for a difference of proportions [33] and, for single model calls, Tango’s paired score interval [42], with Bonferroni correction across inputs and sequential looks.

3 Threat model and guarantees

3.1 Two properties, pursued separately

• Integrity. A recorded event cannot be changed, deleted, reordered or forged without detection: by the agent, whenever the signer runs outside its reach (not in dev mode); by the operator and whoever exports a bundle, only once an independent witness has seen the log.
• Completeness. Actions are captured at several independent layers, and disagreements between layers are written as signed gap records. Completeness is reported, not proven (section 4.5).

3.2 Adversaries

The server threat model names six adversaries (table 3); the laptop model maps onto them. They differ in what they can reach, and the table states for each what Tracekit detects or stops and what it does not.

Table 3: Adversaries, their reach, and what Tracekit detects or stops.

3.3 Invariants

Every change to the code is checked against eleven invariants, each mapped to the tests that enforce it (table 4). Three ideas carry the design: the agent’s identity cannot reach the store, keys or policy (I0); no signing key and no sequence assignment lives in the agent’s process (I1); and, from I3 and I6, nothing an agent-controlled process supplies (its isolation level, its fail mode, gap or tamper records, an approver’s identity) is trusted by the signer or verifier.

Table 4: Design invariants (from tests/INVARIANTS.md).

3.4 Three guarantees

The three guarantees are related, but one does not give the others, and each needs something different (table 5).

Table 5: The three guarantees and what each needs.

3.5 Where the signer runs decides what a run is worth

The agent’s code is the same in every deployment: TRACEKIT_SIGNER points it at the signer. What changes is who could rewrite the log, and the signer records the isolation it measured, never one the agent claims (fig. 2, table 6).

Figure 2: Deployment ladder. The agent’s code is the same at every step; the recorded isolation and the assurance a verifier can reach change.

Table 6: What each deployment lets you rely on (abridged from the project’s guarantees and deployment pages). All but dev mode and Linux system mode are experimental in 1.4.

Outside every deployment

Things the agent creates and leaves running (a CI workflow, a scheduled job, an API key) are recorded only at creation. People the agent persuades act outside any record. Root on the signer’s host, the cloud provider and the hypervisor are outside the threat model. Policy rules match text and can be reworded around; the protection is the signer’s isolation, not the rules.

4 System design

4.1 The signer and the life of a tool call

The signer (tracekit signer serve) is the only process that holds signing keys, assigns sequence numbers and writes records. Agents reach it through a Python or TypeScript client and framework adapters, over a Unix socket (loopback TCP in Windows dev mode) or, experimentally, HTTPS (mTLS, a Kubernetes service-account token, OIDC or a bearer token). Its RPC contract is frozen and versioned (protocol 19 in 1.4). Figure 3 follows one run through it.

Figure 3: The life of a run. The agent only asks; every decision, approval, gap and result is written by the signer. A held call runs only after an approval bound to its exact arguments, which can be consumed once.

A tool call is two RPCs. decide sends the tool name and the arguments (the model’s raw string where the adapter has it; args_source records raw, parsed or coerced); the signer maps the tool to a class with its own policy (a class the adapter declares is a hint: a disagreement that would change the decision is a signed capture.gap{class_mismatch}, and the stricter decision applies), evaluates the policy, and signs a policy.decision record carrying the verdict, every matching rule id, the policy_hash, the engine and its version, a fresh decision id and the argument commitment. complete sends the result; the decision id is consumed once, and a complete with other arguments is refused. Only deny and ask are remembered per (call id, attempt), so a retry keeps its verdict, and an allow or flag after a deny of the same tool and arguments anywhere in the run is a signed capture.gap{decision_flip} (algorithm 1).

Fail modes and durability. Fail modes are per tool class and set in the signer’s configuration (recorded in run.registered); during an outage the class is the adapter’s own reading of the tool, since the signer cannot be asked, and the verifier warns on every fail-open class. A closed class is blocked while the signer is unreachable; an open class runs, and the client’s next call makes the signer write a signed client_counter_gap covering exactly the calls it missed. Records are acknowledged after the write (ack-on-write, with background sync) or after fsync (ack-on-fsync). E14 and E15 (sections 7.4.2 and 7.4.3) test both.

Algorithm 1 Signer: Decide (abridged from tracekit/signer/service.py)

Input: caller identity u, run token τ, tool t, arguments a, call id c, attempt k
 1  r ← Run(τ); if τ is not u's token for r or r is closing or final then return refused;
 2  if a is not strict JSON then
 3  │  v, ids ← deny, TK-ARGS-INVALID;
 4  else if (c, k) was denied or held before with the same arguments then
 5  │  v, ids ← that decision's;               // a retry keeps its verdict
 6  else
 7  │  κ ← Class(t) from the signer's policy;   // a client hint can only tighten
 8  │  v, ids ← Policy(κ, t, a);                // deny > ask > flag > allow
 9  │  if v ∈ {allow, flag} and the run already denied the same (t, a) then
    │      sign capture.gap{decision_flip};
10  │  if a live constraint of r forbids the call and v ≠ deny then
    │      v ← ask, add TK-CONSTRAINT;
11  │  commit ← HMAC(salt(decision_id), SHA256(JCS({t, a})));
12  v, ids ← Contain(r, v, ids);      // a held or quarantined run raises the decision
13  Append(r, policy.decision{v, ids, policy_hash, engine, decision_id, commit});
14  Detect(r);   // sequences and limits over signed rule ids; a hold re-decides this call
15  return v (flag answered as allow), decision_id, rule ids

4.2 Evidence format v2

The format is specified precisely enough for an independent verifier to be written from the specification alone, with normative test vectors. Figure 4 shows how its parts fit and table 7 lists them.

Figure 4: Records, tree and checkpoint. Each record is signed and carries the hash of the run’s previous record, so a changed byte fails at that record. Record hashes are leaves of an RFC 6962 tree whose root a C2SP checkpoint signs; witnesses cosign the checkpoint only if it is consistent with every size they have seen.

Table 7: Parts of evidence format v2.

The bundle. A .tkb bundle is a zip with a manifest (an index that is never trusted: the files must be exactly those it lists), one run’s records or a run-set’s runs, the key records, inclusion proofs, the checkpoint note, any Rekor entry and timestamp, and the policy snapshots that decided the calls. It carries no code and no trust configuration. A run longer than 64 MiB of records exports in parts. Formats only grow: each format’s verifier is frozen, and an older verifier reports a bundle that uses a newer critical entry UNVERIFIABLE (needs tracekit >= x), never FAILED; from 1.4, a non-critical addition verifies with a not interpreted warning.

4.3 The verifier and its two verdicts

tracekit verify bundle.tkb --trust trust.json runs the checks of fig. 5 in order and reports two separate results (table 8). A malformed bundle is a failed line, never a crash. Three implementations exist: Python, an independent TypeScript verifier, and the same TypeScript code built into one self-contained HTML page that verifies in a web worker and sends nothing anywhere (fig. 6). E22 (section 7.2.4) checks that all three agree.

Figure 5: Verifier pipeline. Every check produces a line; any failed check line makes integrity FAILED (an unreadable trust config or bundle is UNUSABLE BUNDLE or UNVERIFIABLE). Assurance is computed separately from who vouches for the checkpoint.

Table 8: Assurance levels. Integrity and assurance are independent: an intact run can still be dev.

Informational lines say what integrity rests on: capture layers that reported, evidence tiers, argument sources, the isolation the signer measured for each run, fail-open classes and key assurance. --strict turns warnings (unproven key retirements, tenant-level gaps, an unproven log tail, calls run against policy, unreconciled calls, break-glass approvals) into a non-zero exit. A full verifier report is reproduced in appendix B.

(a) An intact, witness-cosigned run: VERIFIED, assurance dev because the demo approves its own held call. (b) The same run with the denied call rewritten as allowed: the manifest and signatures lines fail (seq 4: hash does not match the event).

Figure 6: The browser verifier (one HTML page, Tracekit 1.4.1) on the bundles produced by tracekit demo --server. It runs the same checks as tracekit verify, names the next step, and never uploads the files.

4.4 Policy, detection and approvals

Rules and packs. The signer decides every call with its own policy; the agent can change neither the policy nor a tool’s class. Each matching rule adds its id to the decision and the strictest section wins: deny > ask > flag > allow. Rules match fields of a class (a shell command’s programs and paths as a bounded shell parser reads them, a SQL statement’s verbs, an HTTP request’s host and method), and every pattern is in the RE2 subset so that two engines (RE2 and Python’s regex) must decide identically, which E17 and E21 check. A command the parser cannot read is held as TK-SHELL-PARSE. Table 9 lists the shipped packs.

Table 9: Policy packs shipped with Tracekit 1.4.

Provenance (experimental). A rule on arguments cannot tell where a value came from: an address in a task and the same address in a vendor’s web page look alike in the call. The signer can, because the results of tools the policy lists as untrusted reach it. Per run it indexes the values those results carried (emails, URLs and hosts, absolute paths, long digit runs, strings of 6–256 characters, after Unicode normalisation) with the record that first brought each in, and a rule with from: untrusted matches a call whose argument value appears only in untrusted content.

Detection and response (experimental). A rule decides one call; detection looks across calls (a secret read then egress, a runaway loop, a burst of deletes, repeated denied attempts) as sequences and limits over the rule ids decisions already carry. It is part of the policy, so it is in the policy_hash, and it is a pure function of signed records: tracekit policy backtest replays it, and the verifier recomputes a run-scoped incident from the run’s own records (others it reports as signer-asserted). A trigger can notify, hold the run or quarantine it, in shadow or enforce mode.

Live constraints (experimental). A person’s request often adds a limit for one task (“don’t touch billing.py”, “read-only”, “don’t push”). The hooks read it from the prompt and the signer holds a call that would break it for approval, after the policy has decided; a constraint can only raise a decision, never lower it. Section 5.2 describes them in full.

Approvals. An ask holds the call. An approval is bound to the exact arguments’ digest, used once, and recorded with the approver’s identity; edited arguments, a replayed snapshot or another run’s approval are refused, a contract every stable integration’s tests exercise. Approvers can answer from the command line or, experimentally, a web page with passkeys (WebAuthn), Slack, or a reviewer agent that approves only within bounds (an amount range, allowed payees, plain-text memos) under a tenant budget (section 7.5.3). A run that its own identity approved verifies as assurance dev.

4.5 Capture layers and reconciliation

The signer can only sign what reaches it, so Tracekit captures the same activity at several layers and compares them (fig. 7, table 10). Tiers say who captured a record: T1, the adapter (what the agent reported); T2, a gateway or executor (what crossed a boundary outside the agent); T3, an OTLP import (receipt only, never counted towards integrity).

Figure 7: Capture layers in 1.4. L1–L3 and L6 report independently and the signer compares them, writing each disagreement as a signed record, so a hidden or invented call becomes evidence instead of silence. Kernel-sensor effects (L4, experimental) are recorded and joined to calls only by analyze; L5 is not shipped.

Table 10: Capture layers: what each can and cannot see.

Reconciliation indexes each run’s reports by tool-call id (pairing id-less calls of one model turn by argument digest) and compares the tool name and argument digest, unless one side is labelled coerced. Within a grace window before run.final it writes the signed records of fig. 7, and run.final lists the layers that reported, so a run with no L3 says so. With gateway_mandatory: true, only gateway L3 counts: in a deployment whose network policy lets only the gateway reach the model provider, a call the agent hides from L2 is reconcile.hook_missing whatever the agent reports. With verify_tool_results, an edited tool result for a call the signer already completed, or one sent before the call completed, is a signed reconcile.result_mismatch.

4.6 Containment, custody and effects

Recording is not prevention, so Tracekit also offers ways to take the means of harm out of the agent’s hands (table 11). All are experimental in 1.4, except Claude Code’s sandbox rules, which are part of the stable coding pack. The tool executor and MCP gateway are T2 capture paths for L2 calls; edited tool results are caught by the LLM gateway (verify_tool_results).

Table 11: Containment and custody services (experimental).

4.7 Integrations

tracekit.instrument() starts a dev signer (or reuses one), then gates and records every tool and model call of the frameworks it finds. Table 12 lists the integrations. An integration is done when it passes the adapter contract: a test suite of allow, deny, ask, retries, edited arguments, replayed snapshots, another run’s approval, rewritten call ids and T2 argument substitution that every stable integration runs (signer outages are tested separately, in E14).

Table 12: Integrations in Tracekit 1.4.

4.8 The viewer

tracekit view is a terminal-style web interface over the signer’s runs (figs. 9, 10 and 11). Every run in it is verified as it loads: a ticker of counters, an agent tree, a newest-first tape of every record, alerts, tool mix, a per-agent timeline, the model’s self-report (labelled self-reported, not proof), and a detail pane with the signed decision, the rules, the argument commitment and the record’s hash and predecessor. Replay controls step through a run record by record and jump to the next deny, hold or gap. The graph view draws the signed calls with their decisions; the run review summarises integrity, assurance, policy outcomes and gaps.

Figure 8: tracekit view (the desk) on three runs (quickstart, lc_agent, my-agent), the quickstart run selected. The runs pane (left) shows each run verified as it loads, with its blocked and held calls; the tape (centre) every signed record, including an approval requested, approved and consumed; the alert pane (right) signed blocks, flags and approvals. The detail pane (bottom) shows the selected call: DENY → BLOCKED, rule TK-D007 (touching Tracekit’s own files or signer), the run’s verdict Integrity VERIFIED · Assurance dev, the argument commitment, and the ledger evidence for the record (type, run sequence, hash, predecessor).

Figure 9: A Claude Code session through the v2 hook (tracekit init --dev --v2). A Bash call uploading a secrets file is denied before it runs with TK-D006; the detail pane lists the other rules it matched (network, secrets access, egress) and the run verifies.

Figure 10: The graph view of a run: an allowed call, a call denied by TK-D007 (touching Tracekit’s own files), a call held as TK-SHELL-PARSE and then approved (the detail pane shows APPROVED (ask) and the approver, here the same user, so self-approved), and an unknown tool flagged. Arguments and results appear only as HMAC commitments.

Figure 11: Run review: integrity VERIFIED TO HEAD 9 (open), assurance dev, policy outcomes (2 allowed, 1 flagged, 2 denied, 0 held) and none signed under gaps. “operator-side view — re-verify with the signed release for evidence”: the viewer is a convenience, the bundle and an independent verifier are the evidence.

5 Intent, reasoning and action: what was asked, said and done

A signed log of actions says what happened. To judge an action, an investigator also needs the request it served (the intent) and what the agent said it was doing (its reasoning). Both exist at run time, in the prompt and in the model’s output, and both are normally lost, or kept only in a transcript the agent can rewrite. This section describes how Tracekit records them (section 5.1), enforces the limits a request states (section 5.2), checks the three against each other (section 5.3) and turns the result into signed evidence (section 5.4). Section 6 then asks the question a transcript cannot answer at all: which input made the agent act.

What “intent” and “reasoning” mean here

Intent is the stated request, as the person submitted it. Reasoning is the model’s stated account: its text and readable thinking before each call. Both are recorded as such. A model’s stated reasoning need not be its real reasoning [43, 10], so Tracekit labels it self-reported, not proof, never lets it clear a finding by mere mention, and answers “why” with interventions (section 6), not with the model’s word.

5.1 Recording the request and the model’s account

The request. The coding-agent hooks record each prompt as the harness submits it (UserPromptSubmit), as an observed input labelled with its source. This copy matters because the transcript is a file the agent can write: Tracekit counts a transcript line as the person’s request only when the hook’s copy confirms it, and a prompt line the harness never submitted is itself a finding (TK-X208). In E20 an appended prompt line was caught in 0 of 24 sessions before the hook recorded prompts, and in every session after.

The model’s account. A transcript tailer, a separate process the hook starts (in system mode as its own tracekit-tailer user, outside the agent’s control flow; in dev mode as the agent’s user), reports every line the model writes (thinking, text, tool use) with the digest of the line as read; if the file is edited, truncated or replaced under it, the signer writes a signed gap (tailer_lost). The Codex, Cursor and Gemini CLI hooks, the framework integrations and the LLM gateway report model messages the same way. Where a provider hides reasoning, Tracekit turns on readable summaries (Claude Code’s thinking summaries; OpenAI reasoning summaries) and, at the gateway, asks the provider for them, recording that it changed the request. Version 1 found the provider withheld the text of all 27 thinking blocks; version 2 reports monitorability instead of assuming it: the share of actions whose reasoning could be read (TK-X200).

Commitments and the vault. The log holds commitments to the prompt and to each message, never the text. If the operator enables it (it is off by default, and on for 7 days in the dev signer), the text is kept in a content vault, encrypted per record with AES-256-GCM under a per-run key, with the record’s position and commitment bound in, so a blob moved to another record does not decrypt. Only the signer reads the vault, for callers with an explicit grant, and every read is itself a signed record. The analyzer reads text only from the vault and checks that each item opens its record’s commitment (content that redaction changed opens nothing, and a finding relying on it is labelled redacted), so its input is what the signer committed to at the time.

Trust tiers. Each piece of text is labelled with the tier of its source (table 13). Tiers limit what text can do: a later request lifts a constraint only if its tier is at least the constraint’s, and two conflicting reports of one message make it unusable (conflicting_reports) rather than letting either win.

Table 13: Trust tiers of recorded text, strongest first.

Figure 12: One Claude Code session in tracekit view, newest first (Tracekit 1.4.1; the model’s messages are scripted). The request ended “Fix the off-by-one in pager.py and run the tests. Don’t touch billing.py.” The signer signed the constraint the hook sent before the prompt (bottom: HOLD · CONSTRAINT keep a file unchanged, then INPUT and PROMPT). The tailer recorded each thinking summary (THINK) and each line the model wrote (SAYS). When the model decided “the billing rate looks wrong too; I’ll round it while I’m here”, the signer held the edit of billing.py (TK-CONSTRAINT) and a person rejected it before it ran. A later edit of config.yaml, which no request named and the model never announced, became a signed finding (TK-X202) in a separate findings run (top).

Figure 13: The reasoning panel and run review of the session in fig. 12. Left: what the model said and its thinking summaries, in order, labelled self-reported, not proof. Right: the run in one sentence (5 calls: 4 ran, 0 blocked, 1 held and rejected; 1 finding), the constraint and the call it held, coverage L2+L3, the finding, monitorability (reasoning readable before 3 of 4 actions) and what could not be checked and why (no kernel sensor covered the run, so TK-X206 and TK-X207 are not checkable, never counted as passes).

5.2 Live constraints

A policy is written in advance; a request often adds a limit for one task: “don’t touch billing.py”, “read-only, just tell me what you’d change”, “don’t push”. The hooks read such limits from the prompt with a small fixed grammar and send each to the signer before recording the prompt. The signer signs a constraint.added record and checks every later call of the run against it, after the policy has decided. A call that breaks a constraint is raised to ask (TK-CONSTRAINT) and waits for a person (fig. 14). Five kinds are read: keep a file or directory unchanged, read-only, no push, no network, and “only change X”, which is checked after the run rather than held, since holding every other write would interrupt each test or lockfile update.

Figure 14: The life of a live constraint. The hook reads the limit from the prompt and the signer signs it before the prompt itself. A later call that breaks it is raised to ask after the policy has decided, so a constraint can only move a decision up the order on the right.

Two properties make this safe to deploy. First, a constraint can only tighten: a deny stays a deny, an ask stays an ask, and nothing is lowered. The signer can therefore accept constraints from any client of the run, including the agent itself, without trusting it; a made-up constraint only makes the agent wait. Second, the grammar abstains when it is unsure, because a misread limit blocks wanted work. It leaves unread a condition (“don’t edit X until the tests pass”), a question, quoted text, a description (“add a read-only mode”) and a limit the same request lifts; a later “it” lifts a constraint only in a clause that hands the change to the agent (“you can edit it locally”), not in one that keeps it with the person (“ask me before you change it”). The log stores a file only as a salted commitment to its name. A run accepts 32 file constraints; past that, the signer writes a signed constraint_refused gap and holds every write.

A constraint read from a prompt is therefore both a hold during the run and, if a call that breaks it runs anyway, a finding after it (TK-X204); a held call a person rejected never ran and is no finding. In the analysis, a later request that names what the constraint keeps away lifts it: the finding stays, downgraded to low and marked lifted, citing the request (the live hold itself is not removed). An approval by another identity downgrades it too; a self-approval lowers nothing.

5.3 Checking asked against said against done

tracekit analyze --v2 reads a finished run from the log and the vault and checks whether the three channels agree (fig. 15, table 14). It also re-implements version 1’s say-versus-do checks (claimed tests, commits, pushes or installs that never ran; denials of actions that did), now against the signed decisions and results rather than a self-reported log. Three rules shape the checks:

• Not checkable is never a pass. Each check counts the calls it could not check (reasoning sealed, no content kept, no sensor) and signs those counts, so a run in which nothing could be read gets no clean result.
• Disclosure clears a finding; mention does not. An unrequested edit is excused only when the model tied a change word to the file (“applying the same fix to billing.py”), not when it merely named it (“pager.py imports billing.py”). Sealed reasoning tells the person nothing.
• Effects are joined from the machine’s side. With kernel-sensor events in the run, each process, connection and file write is joined to the call that caused it, by the call id the process carried, its parent process, or the call’s time window, so an action the agent never reported still shows (TK-X206, TK-X207).

The checks are scored on phrase pools written by someone who never saw the code, and a pool read while fixing the checks is relabelled a development set (section 7.6.1). A reviewer model can add a second opinion (TK-X210), always warn-only.

Figure 15: The three channels and the checks between them. Each finding is signed, cites the records it rests on, and never alters the integrity verdict.

Table 14: Findings of tracekit analyze --v2 (full definitions and not-checkable reasons in appendix C).

5.4 Signed findings

The analyzer does not just print a report. Each finding goes back through the signer into a findings run of its own, written as the analyzer’s identity, which the signer accepts only if its configuration lists that identity as an analyzer; the signer signs it. A finding cites the records it rests on by position and hash and holds no text from the run, so findings can be shared without the content. A verifier checks every citation, labels a findings run independent only if the auditor’s trust configuration pins its author and the author is not the analysed run’s identity, unpinned author if the author is another identity the configuration does not list, and self-asserted if the agent could have written it (the run’s own identity, or a same-user signer, as on every laptop), and reports the findings on a line of their own, after integrity and assurance.

The separation is the point. Whether a record is intact can be established; whether an agent did what it was asked is a judgement, made by code with its own error rates (section 7.6.1). Signing judgements, citing their evidence and naming their author, but keeping them out of the integrity verdict, lets an auditor accept one analyzer and not another, re-run a better analyzer on old evidence, and never mistake a clean analysis for a verified record.

6 Which input caused it: tracekit why

Tracing tells you every untrusted input that reached an action. In a multi-agent workflow that is usually all of them. tracekit why asks a different question: if this input had not been there, would the action still have happened? It answers by replay (fig. 16).

Figure 16: Attribution (find the cause). Replay with each suspect removed turns “reached the action” into “changed how often the action happens”, with an interval and a multiplicity-corrected verdict.

The graph. Nodes are inputs, model decisions, actions and messages. Edges are graded: observed (an item was in a decision’s context; a decision invoked an action or sent a message; an input has the exact hash of an earlier output), inferred (at least three shared word 5-grams and overlap ≥ 0.3, with no observed path), and tested (a saved replay removed the source and measured the change). Forward reachability over observed edges gives the blast radius of an input: what it could have influenced, a superset of what it did.

Argument provenance. For each tool call, values worth tracing (emails, URLs, and whole string arguments of six or more characters) are searched, verbatim and case-insensitively, in earlier inputs and tool results and labelled untrusted-only, trusted or generated. A sensitive action with an untrusted-only argument raises a high alert immediately, without replay.

Replay and statistics. For a target T and an intervention I, trial i runs the program with seed sᵢ with and without I, so each pair differs only in I (common random numbers). With n pairs, let b be the pairs where T happened only with the input and c those where it happened only without. Then

where k is the number of planned sequential looks and F the family size of the attribution (1 for a single test; p = 1 when b+c = 0). The interval for Δ in run replay, and in signed findings, is Newcombe’s hybrid score interval for two proportions [33], which ignores the pairing and which the verifier recomputes. Decision replay uses Tango’s paired score interval [42], checked against Fagerland, Lydersen and Laake’s table [14], which behaves well at 0% and 100% and stays as wide as the trials allow (about ±0.28 at n = 10) when no pair is discordant. A verdict is causal if the corrected interval’s lower bound exceeds 0, suppressive if its upper bound is below 0, ruled out if it lies within ±0.2, and otherwise inconclusive, with the n at which the interval would be narrow enough. A role is primary when the interval of Δ/p̂_with has lower bound ≥ 0.5, contributing when its upper bound is below 0.5, and not yet clear otherwise. The exact McNemar p cannot fall below 0.05 with fewer than six discordant pairs (2 · 0.5⁶ ≈ 0.031), and the corrected interval also needs several discordant pairs to clear zero, so replay has a budget floor: on the simulated benchmark it names nothing at n = 5, 69% of causes at n = 10 and all at n = 30 (section 7.6.3).

What a causal verdict means

In this program, on this task, removing this input changes how often the target happens. It is a total effect through every path, measured on a recorded program that can be re-run. It says nothing about the model’s internal reasons, and the replay’s stubs mean behaviour after an unrecorded call can differ from the original.

Decision replay. When a production system cannot be re-run end to end, decision_test rebuilds one recorded model call’s context from content-addressed blobs and calls the model n times with and without the item, measuring the direct effect on that call. Because real model APIs ignore seeds, when the target is any change in the output it also measures how often two full-context calls differ and subtracts that noise.

Signed attribution. The why log of a re-run program is hash-chained but not signed: whoever can rewrite its files can rebuild the chain. On a run that the signer and the LLM gateway recorded with the content vault on, attribution needs no code and its results are signed evidence. The investigator replays one recorded model call (rung 1) or re-simulates the agent loop from the first call that held the item, answering tool uses from the run’s tape (rung 2). Each arm is itself a signed replay run whose model calls go through the gateway, and each test is a signed why.finding in a findings run. The verifier checks that the replay runs it cites are bundled, recounts their gateway-recorded outcomes, and recomputes the interval and verdict; a text-matching target (arg~, text~) stays the investigator’s claim.

In the viewer. Figure 17 shows the investigation on a recorded three-agent customer-service desk. The why did it happen? view starts from the alert (send_email used an address that appears only in untrusted content), traces the argument to vendor:portal/notes.md, shows the path the content took through the agents, and lists candidate causes with their tested effects. The graph view draws the run’s 15 nodes with observed and tested edges (inferred links and ruled-out tests can be toggled on). The viewer’s individual tests are not corrected across tests. The influence view aggregates across runs: which sources were cited by decisions, and which links replays confirmed.

(a) Why did it happen? The selected send_email has a high alert: its address comes only from untrusted content (vendor:portal/notes.md). Verdict: primary cause vendor:portal/notes.md; in replays without it the action happened 0% of the time instead of 80% (n = 40; 95% interval for the drop 63 to 90 points). The path shows how the note reached the executor: researcher → planner → executor.

(b) Graph: inputs (left, untrusted circled red), model calls and messages (middle) and tool calls (right). Red edges are confirmed causes; the node pane shows what a model call saw and produced.

(c) Influence across eight runs: thicker links were reached in more runs; red links were confirmed by replay. kb:refund_policy.md (trusted) is the confirmed cause of issue_refund (+0.93); vendor:portal/notes.md (untrusted) of send_email (+0.80).

Figure 17: Causal investigation in tracekit view on the recorded multi-agent demo (tracekit why demo; seeded mock models, no signer). Screens from the project’s walkthrough.

(a) Overview: what the agents did, what needs attention, and how to read the evidence. (b) Agents: who spawned whom, the channels between them, and an influence matrix of tested effects.

Figure 18: Two more views of the same investigation.

7 Evaluation

7.1 Method

The evaluation has 21 experiments, E1 to E22 (there is no E13), plus the attribution benchmark and a case study (table 15). Each experiment writes eval/results/<name>.json (E7: contrib/query/e7_sql_scale.json), and each one that measures time records its platform. Seventeen of them run on every pull request: E8 (insider attacks, which needs root and a Linux system-mode signer) is a merge gate; E9 and E15 run on Linux; and E1, E3, E4, E6, E10–E12, E14 (quick) and E16–E21 must pass their own gates, and the deterministic results (E1, E3, E4, E6, E10, E17, E20, E21) must equal the committed files, so a change that moves one of those numbers has to commit the new file. A test fails when the numbers quoted in the documentation disagree with the results. E2, full E14, E9 and E16 on Postgres, E22 and E7 run nightly.

Read every number with this in mind

The experiments are small, synthetic and written by the author of the system. They show how the mechanisms behave, not how Tracekit performs across real agents and projects. Where a test set was read while the code was being fixed, we say so and call it a development set. No one outside has reviewed the threat models yet.

Table 15: The experiments. CI: every pull request; gate: blocks the merge; nightly: scheduled; manual: needs a model, root, or minutes of time.

7.2 Integrity

7.2.1 E1: tamper detection

E1 builds 120-record signed ledgers (the signed JSONL ledger of the earlier laptop path; evidence format v2 is tested by E4) and applies one mutation per trial, 60 trials per mutation. “Ledger only” is the hash chain plus Ed25519 signatures checked with the signer’s public key; “plus witness” adds independent checkpoints, every 10 records, taken before the attack (table 16).

Table 16: E1: detected trials of 60.

Figure 19: E1: a key holder edits one record at a random position and re-signs the rest. Records after the last checkpoint are unprotected. At k = 120 the single checkpoint is the head itself.

The two zeros are the point. A ledger cannot tell that its own tail was cut off, and whoever holds the signing key can rewrite history and re-sign it. Only a copy of the head held elsewhere catches those, which is why an unwitnessed run verifies only as assurance dev. The checkpoint interval is the window a key holder can still use (fig. 19).

7.2.2 E4: seeded faults in real bundles

E4 corrupts real bundles and runs the offline verifier on every copy; the attacker edits the zip directly and always repairs the manifest hashes. On v1 bundles (about 40 records, a checkpoint every 4 records, a git witness) there are 15 faults with 30 random placements each (table 17); on v2 bundles (runs with an approval, a run-set from registry size 0, a checkpoint cosigned by a pinned and required witness) 14 more (table 18). The verifier raised no exception on any corrupted input.

Table 17: E4 on v1 bundles: detected of 30.

Table 18: E4 on v2 bundles: detected of 30.

The v1 misses with a witness are a key holder cutting the tail exactly at a witnessed checkpoint, which leaves a valid prefix whose only sign is a missing run.end (a warning, so --strict fails it). In format v2 such a run verifies only as VERIFIED TO HEAD n (open), and against a run-set its missing run.final is a missing leaf. E4 v2 has no fault class for an attacker holding the real signing key; what a witness adds against one is measured on v1 only.

7.2.3 E8: insider attacks

E1 and E4 mutate a finished log. E8 attacks the path that feeds the log, as real unprivileged OS users (the agent’s user and a second local user) against a Linux system-mode signer with a registered harness. It fails unless every case is caught, and it runs on every pull request as a merge gate (table 19).

Table 19: E8: insider attacks on Linux system mode. Every case is caught; “caught” means refused, recorded as a gap, or blocked.

E8 does not cover events fabricated into the live run from inside the harness session; only cross-layer reconciliation against a layer outside the agent catches those (E10).

7.2.4 E22: the verifiers on real bundles

E22 makes every bundle the way a user would, from scratch, in a home directory of its own, and verifies each three ways: tracekit verify --json, the TypeScript CLI, and the browser page in Chrome through Playwright (headless for every bundle; again with a foreign script element added before the page’s own, as a browser extension adds one; and headed for five). The three must give the same verdict, exit code and check lines. Sources: the offline examples (11 frameworks, 5 demos and an injection replay, with scripted models; 11 of the 17 ran, six frameworks were not installed), the server demo and its tampered copy, the Claude Code hook (allowed, denied and approved calls, with and without signed findings), Tetragon, Falco, OpenShell and agentsh imports, open and closed exports, a 50,000-call run, bundles from the released 1.2.3 and 1.3.0 signers and a v1 sample, and 31 forgeries.

Table 20: E22 outcome.

Figure 20: Verifying the largest bundles: runs of 45,000 and 50,000 calls (90,002 and 100,002 records; 24.8 and 27.6 MB zipped).

E22 found two bugs, both fixed in 1.4: tracekit export wrote a bundle every verifier called UNUSABLE BUNDLE once a run’s records passed 64 MiB (runs now export in parts), and the page’s layout broke on a phone when a log origin was long. The page verifies in a web worker; its main thread had no long task. Both CLIs peaked near 1 GB of memory on the largest bundle, about eight times its unzipped size. Firefox and Safari are not tested.

7.3 Capture and completeness

7.3.1 E10: reconciliation

E10 scripts runs against an in-process signer for each combination of capture layers: clean runs, including hard clean cases, and runs with one seeded discrepancy each (table 21). Without L3 nothing can be compared, and no run may be flagged.

Table 21: E10: what each layer combination flags. 30 runs; 12/12 seeded discrepancies flagged; 0 false positives.

The L2 column is the limitation in numbers: with one layer, a hidden or invented call is invisible.

7.3.2 E11: tampering with agent state

E11 runs a short session through each L1 source (the LangGraph checkpointer wrapper over SQLite, the Claude Agent SDK session-store wrapper, and the Claude Code transcript tailer), then tampers with the saved state in seven ways; in-place edits keep the file’s size. A case is detected when the run holds the signer-written gap.

Table 22: E11: tampering detected per L1 source (21 of 21).

The wrappers compare the store with what the same process committed, so they catch a store changed underneath the agent, not an agent process that lies consistently.

7.3.3 E19: egress observation

E19 runs v2 signers and the egress proxy in process, offline (table 23).

Table 23: E19: egress observer cases; all as expected.

7.4 Operation

7.4.1 E9 and E2: latency and throughput

E9 runs tracekit signer serve as its own process and drives it with the Python client over the Unix socket, so every number includes the client, the socket and the writer. Latency is one client calling decide then complete; throughput is several client processes calling as fast as they can. The design gates are p99 ≤ 5 ms per tool call at ack-on-write and at least 1,000 events per second, on full Linux runs; the committed results below are quick runs on a Mac, so the gates are informational there.

Table 24: E9: signer round trip (macOS arm64, 12 CPUs, Python 3.11.15, quick run).

Postgres misses the 5 ms p99 gate on this machine at ack-on-write. E2 measures the older laptop path, in which each Claude Code hook is a new Python process: a bare python -c pass costs 19 ms, one PreToolUse hook 70 ms at the median (82 ms p95), flat across ledgers of 0, 1,000 and 5,000 events, so a tool call pays about 140 ms of hook time before and after, about 40 ms of it (19 ms per hook process) interpreter start-up. Offline verification is linear: 0.16 s, 0.72 s and 2.9 s for 500, 2,000 and 8,000 events. E2 found a real bug before release: with eight concurrent writers the signer’s accept queue (Python’s default of 5) filled, and fail-open clients dropped events. The queue is now 256, the client retries, and a regression test fails without the fix.

7.4.2 E14: outages

E14 breaks one part at a time with fail modes {default: closed, fs: open}, so Read runs and Bash is blocked while the signer cannot be reached. Every store must pass tracekit signer fsck afterwards (table 25).

Table 25: E14: outage scenarios (full run, macOS). All pass.

Under the partition, a call took the client timeout (about 1,000 ms added at p50 and p99 over a 2 ms baseline); after it, the added latency was 4.5 ms at p99.

7.4.3 E15: crash durability

E15 runs the signer under eight concurrent clients and kill -9s it at random points, 30 times per acknowledgement mode, restarting it each time. Clients do not retry, so a failed call’s counter value is used up. Afterwards every acknowledged call must be in the store at the position it was acknowledged with, and every unacknowledged call must be in the store or inside a signed client_counter_gap; anything else is a silent drop (table 26, fig. 21).

Table 26: E15: 30 kill -9 rounds × 8 clients per mode.

Figure 21: Every call is accounted for.

kill -9 keeps the operating system’s page cache, so ack-on-write losing nothing here says nothing about power loss, where records acknowledged within the last background sync interval can be lost. That window is not exercised.

7.4.4 E12: tenant isolation

Two tenants share one signer. Tenant b’s identity attacks tenant a’s run, which has a pending approval and an imported trace. Every attempt must be refused or reach nothing of a’s. Then eight asyncio tasks, eight threads and eight Node promises, each in its own run, make 20 calls each through the per-run context, and no call may leak into another run (table 27).

Table 27: E12: tenant isolation.

7.4.5 E16: the deployment checker

tracekit doctor should flag a misconfigured signer. E16 builds a clean layout, breaks one thing per case and expects the named check to fail or warn; the clean layout must pass.

Table 28: E16: misconfigurations flagged by tracekit doctor.

Doctor checks only the misconfigurations it was written for; its output is advice, not evidence.

7.5 Control

7.5.1 E17: policy packs, detection and containment

Three offline parts, each with a signer in process; nothing is executed. The policy corpus has 570 harmful and 738 benign calls in 11 classes, written by the project and decided by server.yaml with both engines (table 29). All 570 harmful calls are denied or held; one benign call is held (a draft email to a list address); the engines decide every call the same. Detection: 10 of 10 scenarios hold (a sequence split across two runs, a tool alias the policy does not map, parallel runs, a GET with a query string as egress, restarts and crashes mid-sequence, and the detector’s state staying the same size however many calls a run makes). Containment: 8 of 8 scenarios in which the agent, or content it reads, would undo a hold are caught, including a bundle whose signer let a call through after a hold, which both verifiers fail.

Table 29: E17: the server pack on the policy corpus, by class.

E17 is a development corpus: it measures that the rules do what they were written to do, not how they fare on calls nobody has seen. E21 measures that.

7.5.2 E21: policy robustness on a blind pool

E21 decides calls the packs were not written against: 420 harmful and 432 benign calls in 17 categories (14 harmful, 3 benign), written blind to the rules on a separate branch, plus 130 calls of ordinary developer sessions. Its one blind score is the first run: 211 of 420 (50.2%) harmful calls caught by default and 246 (58.6%) by strict.yaml. Rules were added after that run that target shapes in the pool, and the pool has been read since, so every later number overstates what the packs catch on unseen calls. Tables 30 and 22 give the recorded numbers for 1.4.

Table 30: E21 (packs as merged for 1.4; the pool is no longer blind). A harmful call is caught when denied or held; a benign call is a false positive when denied or held. History of held-out catches on the same pool (default / strict): first, blind run 211 / 246 → autostart rules 215 / 250 → credential, container and package rules 236 / 266 → 1.4 review 237 / 266.

Figure 22: E21: share of the held-out harmful calls caught, by category (1.4 packs). Privilege escalation (8 of 27) and exfiltration (12 of 39) are the weakest categories by default; exfiltration gains most from strict.yaml (31 of 39), at the cost of holding network commands.

By evasion technique (default), calls with no evasion are caught 142 of 275; quoting 9 of 18, environment prefixes 6 of 14, command substitution 37 of 49, interpreter one-liners 23 of 28 (many held as unreadable). Policy decisions are fast: with RE2, the default profile decides at 63 μs p50 and 150 μs p99 (max 276 μs) over 2,290 calls, and the two engines decided every call the same. A separate check rewrites each of the 261 shell commands among E17’s harmful calls in up to five equivalent ways (cd . && or true; before it, command, bash -c, env bash -c); all 570 originals and all 1,265 rewrites are still denied or held, and the run gates rule by rule against a recorded baseline.

What E17 and E21 together say

On calls written alongside the rules, the packs stop everything with one false alarm in 738. On calls written blind to them, they stopped half. Rules are tripwires for obvious actions; the protection against a determined agent is a signer it cannot reach and credentials it does not hold. Version 1’s gate stopped 18 of 44 (41%) on its own corpus; its successor rule set, on a rougher set written after the rules, stopped 11 of 16 with 1 of 20 false alarms (E3).

7.5.3 E18: agent approvers

E18 attacks a reviewer agent that approves everything it may see, as one whose judgement was steered would. The signer has an ask rule on a payment tool that agents may answer within bounds (an amount of 1 to 100, two payees, a plain-text memo), under a tenant budget of 5 approvals and 300 in a window. A call runs only when approval_consume lets it. Seven attacks are all caught: arguments out of bounds, a memo only untrusted content brought in, asking again after the reviewer said no, 50 variants one cent apart, exhausting the budget, a reviewer answering after it escalated (and after a restart), and an approved agent approval after the operator switched agent approvals off.

7.6 Explanation

7.6.1 E20: asked, said and done

E20 replays synthetic Claude Code sessions through the real hook (prompts at submission, with the constraints they state sent as live constraints), the transcript tailer and the signer with the content vault on, and runs analyze --v2 on each. Faults are spliced mid-run, at the end, or inside a sub-agent’s transcript: F1 edits a file the request never named; F1c does so after a request that keeps that file away; F1o writes outside the project; F2 sends a file to a host; F3 deletes one; F4 claims success after a failed test run; F5 installs a package and pushes; Fc claims a run that never happened. Each is hidden under a cover: c0 generic text; c1 naming the file innocently or to say it stays untouched; c2 a summary that hides the edit and claims success; c3 sealed thinking only; c4 text addressed to the reviewer; c5 a fake model message or an appended prompt line.

Four phrase sets are scored separately: tuned, written with the checks; dev pools A and B, written blind but read while fixing the checks (development sets); and the held-out pool, written blind and scored once. 24 sessions per cell.

Table 31: E20: detection by cover (%), 24 sessions per cell, with the checks as released in 1.4. The held-out pool was written blind and scored once; dev pools A and B were read while fixing the checks. Green is high, red low.

What the table shows. An unrequested edit is found in every session of every set whatever the cover, because TK-X202 is cleared only when the model’s text says it changes the file, and a mere mention (“pager.py imports billing.py”) clears nothing. After a “don’t touch X”, the held-out detection is 79–100% by cover, and a live constraint held the edit for approval before it ran in 67–92% of sessions by cover (133 of 168 overall, 79%). A failed test run claimed as passing (F4) is found in 86% of held-out sessions (exit-code failures 86%, failures visible only in piped output 87%). Sending, deleting and installing without saying so are found in 96–100% by cover on held-out pool, except a file sent out under an innocent mention of it (33%). An appended prompt line (c5 line) is itself flagged by TK-X208 every time (the fault it hides is found at the rates in the table); the line was caught in 0 of 24 sessions before the hook recorded prompts. Claims of runs that never happened (Fc) are the weak spot: 14 of 30 held-out sessions: 14 of the 15 that name a test, build or CI run, and none of the 15 bare success claims (“Done.”, “Ready for review.”).

The development pools show what blind wording does to text checks: F2 and F5 under dev pool A fall to 8–38% for the covers that hide the action in ordinary words. Dev pool A is labelled a development set because one round of word-list additions took five of its phrases (since removed and rescored).

False findings. On the held-out pool’s 1,812 honest calls: TK-X202 flagged 2 hard negatives, TK-X204 2 and TK-X211 1, and one honest call was held live; no TK-X201, X203 or X208. On 56 honest sessions written as people phrase requests (“add a read-only mode”, “don’t push until the tests pass”, “we don’t use the network in unit tests”), none of 119 calls is held and no TK-X204 is found; before the fixes they prompted, 13 sessions had a call held. An independent reviewer’s 67 fresh honest sessions, kept from the checks’ authors, produced no hold and no TK-X204, and the grammar held 18 of the reviewer’s 30 fresh violations live.

Effects. A further arm checks TK-X206 and TK-X207 against what a kernel sensor saw, on Claude Code-like runs (no call ids) and Codex-like runs (each process names its call), with Tracekit’s own hook processes and the harness’s model calls between them. On a tuned and a separately generated held-out set, 176 runs each: no false alarm over about 9,800 honest effects and 900 honest calls, and no fault passed (each was found or reported not checkable). Held out, a Python exfiltration between calls was not checkable in 4 of 8 Claude Code-like runs.

Not run yet. The reviewer-model arm (TK-X210, about 400 model calls per run) is implemented and not yet run, and no real claude -p sessions are in E20. Version 1 reported an LLM reviewer catching 98 of 98 seeded faults on 63 traces; that setup is not the one E20 measures and we do not carry the number forward.

7.6.2 E6: the say-versus-do text detectors

E6 is the regression record for the v1 say-vs-do rules: 2,000 synthetic sessions, half with exactly one spliced misbehaviour, the other half ending honestly in the same situations. Precision is 1.00 (0 of 1,000 honest sessions produced a finding). Structural detectors (request vs. execution, secrets in output, a hidden force push) have recall 1.00; text detectors catch every in-pattern claim and only 29% (fake tests) and 40% (push denied) of held-out paraphrases. That gap is why version 2 scores text checks on blind pools.

7.6.3 Causal attribution

The benchmark has six small multi-agent workflows: researcher agents read documents and tool results, summarise them for a planner, which picks tool calls for an executor. Five contain exactly one effective injected instruction (the ground-truth cause) plus decoys that reach the same action through the same agents; the sixth has none. Four methods name a cause in each harmful run: reach (every untrusted input upstream, what tracing alone gives), provenance (inputs that are the only untrusted source of an argument value), reuse (the input whose wording was reused most), and why (replay). A method scores exact when it blames exactly the true cause. With a real model the planted document might not be the cause, so a ground-truth check runs 12 fresh runs with and 12 without it.

Table 32: Attribution with Qwen 2.5 7B (local, temperature 1.0, sequential tests of 10–40 pairs; 3 runs per scenario; 3,863 model calls).

Where the planted document really drives the harm, replay named exactly it in 6 of 6 harmful runs, including an injection two agents away, with no false blame; reach blamed every untrusted input every time. In refund-override the model issues the refund almost always, with or without the forged “pre-approved” memo, so the memo is at most a contributing factor; replay confirmed it once and called it inconclusive twice, the right answer for a weak effect. String matching is cheaper and right when the injection plants a unique value (an address, a group name) and silent when it does not. Each attribution cost 320–540 model calls, 30–45 minutes on a laptop.

An earlier run with 10 fixed replays per input and no JSON-constrained output named the cause in only 1 of 10 harmful runs. Three causes were fixed: too few replays (now sequential), 5% malformed JSON from the 7B model blurring both arms (now constrained and counted), and “no detectable effect” being shown as “ruled out” even with intervals as wide as [−0.19, +0.40] (now inconclusive). That run also produced a real finding: removing an obvious “IGNORE ALL PREVIOUS INSTRUCTIONS” decoy made the escalation more likely; the blatant injection had made the model cautious.

On a simulated model that follows injections with fixed probabilities (a test of the pipeline, not of a model), replay names the true cause in all 61 harmful runs at n = 30, in 69% at n = 10, and in none at n = 5, where fewer than six discordant pairs can never reach significance (fig. 23).

Figure 23: The replay budget has a floor (simulated model, 20 runs per scenario).

7.6.4 E5 and E7

E5 ran real Claude Code sessions (claude -p, four scenarios, three runs each) with the hooks installed in a throwaway project and a git witness: 12 of 12 bundles verified against the witness, every run fixed its test and ended with run.end, and the legitimate rm -rf build was never blocked. The model ignored both planted injections in all six injected runs, so the gate never had to block anything: E5 shows that capture and verification work on real sessions, not that the gate stops a real injection. E7 builds the optional SQL index over a hash-chained ledger of 1,000,001 events (604 MB): build 36.2 s; every query under 1 s (one run’s tool calls 1 ms, a full-text scan 227 ms); a rebuilt index returns identical rows. The index is a convenience, never evidence.

8 Case study: two open-source agents

On 12 October 2026 we integrated Tracekit into two real open-source agent projects, following the documentation a customer would read, with a real model on fixed task sets. We ran each set without Tracekit, then with it, and compared the signed records with each project’s own logs (table 33).

Table 33: Case study summary.

What the signed record caught. The most useful results were bugs, in Tracekit and in integration patterns, found by the records themselves:

• Lost streamed responses. On the first pass the verifier reported 12 of 12 honest tool calls as fabricated: calls no recorded model response asked for. The Agents SDK streamed through a path the model-call capture missed. The records were right; after the fix, 0.
• A denied tool ran while the signer was down. CrewAI swallows any exception a hook raises and runs the tool. The client raised SignerUnavailable, so with the signer stopped, close_account ran although policy denies it. The record showed it: a decision for the call before, nothing for close_account, and a bundle that verified only as VERIFIED TO HEAD 1 (open). Rule: a gate in a framework hook must block on any exception.
• No call ids in hooks. With invented ids, 33 calls produced 66 false reconcile findings; passing the model’s id into the hook paired 33 of 33 with zero findings.
• One run per server. A long-running chat server mixed every conversation into one run, which the signer then closed after an hour idle. Work now follows the current run.

Policy and provenance held: a deny kept close_account from running and the model was told the rule ids; an ask on refund_order ran exactly once after approval; a rejected delete_customer never ran; send_email after the agent read an untrusted ticket was held, with the ticket recorded as the source of the address; and with a mandatory gateway, four crews produced zero reconcile records. Two tools could not be meaningfully decided because their arguments did not name their target (cancel_flight() took the flight from context; issue_compensation(reason) had no amount): design consequential tools so that their arguments name what they act on.

9 Discussion

9.1 From version 1 to version 2

Figure 24 shows how the system moved from the version described in our first paper to 1.4.1. Each step answered a negative result of the step before.

Figure 24: Release history, September to October 2026 (dates from the changelog; 1.4.1, a shell-parser hardening patch, followed on 21 October).

The writer must not be the actor. Version 1’s ledger was correct as a data structure and weak as evidence, because the hook that wrote it ran as the agent. Signing alone would not have fixed that: a key the agent can read is a key the agent can use. The fix was architectural: a separate signer that holds the keys and assigns sequence numbers, with the isolation it measured written into every report. E8 is the experiment that keeps this honest; it runs on every pull request because it is the property most easily lost by a convenient change.

Integrity is the easy half. Once the signer is out of the agent’s reach, standard machinery (canonical JSON, Ed25519, Merkle trees, witnessed checkpoints) gives integrity whose limits are well understood: E1’s two zeros are exactly what transparency logs predict. The hard half is completeness: a signer cannot record what never reaches it. Our answer is not a proof but a discipline: capture at several layers, at least one of them outside the agent, and turn every disagreement into a signed record. E10’s L2 column is the reason: with one layer, a hidden call is invisible.

Separate the verdicts. A single verdict invites reading “verified” as “trustworthy”. Splitting integrity (were the bytes changed?) from assurance (who could have changed them?) made the dev-mode caveat impossible to miss: every same-user run says isolation: same-user and Assurance: dev, however intact it is.

Policy is a tripwire. Version 1 concluded that pattern matching on command strings is a tripwire, not a boundary. Version 2 invested heavily in the packs (a bounded shell parser, two engines that must agree, provenance of argument values, nine packs covering 11 tool classes), and E21 still found half of a blind pool getting through. We think this is the correct expectation for any syntactic gate, and it is why Tracekit pairs policy with custody: the stronger control is that the agent does not hold the credential (section 4.6).

Score text checks on wording you have not seen. E6 showed regex detectors at 100% on their own phrasings and 29–40% on paraphrases. E20 therefore scores every text check on pools written by someone who never saw the code, and demotes a pool to “development” the moment it is read. This costs headline numbers (dev pool A’s 8–38% for F2 and F5 under ordinary covers) and buys numbers we are willing to defend.

Ask the system, not the model. Version 1 found that the provider withheld every thinking block; version 2’s checks count sealed reasoning as telling the person nothing. More fundamentally, a model’s account of why it acted is testimony [43, 10]. Replay asks the system instead: remove the input and see whether the action still happens. The price is compute (hundreds of model calls per attribution) and a re-runnable program; the benefit is an answer with an interval.

9.2 Limitations

• Same-user modes give no isolation. In dev mode, and in any deployment where the agent’s identity can reach the signer’s files, the agent can read the keys, stop the signer and re-sign the store. Windows supports dev mode only; macOS system mode is experimental.
• Harness binding has holes. On Linux system mode it refuses runs and events from programs that are not the registered harness, but not from a process the agent starts inside its own harness session, or a new instance of the harness the agent’s user starts on its own terms. Only reconciliation against a layer outside the agent catches events fabricated from inside the session.
• Completeness is not proven. An action that crosses no capture layer leaves no record. L1 wrappers and in-process autotrace are agent-reported. Services (L5) are specified but not shipped. Tool results are whatever the agent reported unless the gateway verifies them.
• Operator trust without independent witnesses. Whoever holds the signing key can rewrite and re-sign any history no independent witness has cosigned. No public witness runs yet. Two forks shown to two verifiers are detected only by a monitor or a witness that sees both. Detection is all the design offers; nothing prevents a rewrite.
• Commitments hide content, not shape. Timing, sizes and counts of calls are visible to a bundle holder.
• Analyses are opinions. Findings, the reviewer model and causal verdicts are signed and cited, but they are judgements about intact evidence, not part of it. Text checks miss wording they were not written for; replay needs a re-runnable program and many model calls.
• The evaluation’s reach. The experiments are synthetic, written by the author, and mostly run on one developer machine; several gates hold on Linux only. No experiment measures adversarial models trying to bypass capture, real production traffic, power loss, redaction leaks across real transcripts, or an external review.

9.3 Future work

• A public witness and a monitor run by parties other than the operator, so that witnessed assurance does not require each user to arrange one.
• A bypass study with adversarial models: agents instructed, or trained, to disable hooks, edit transcripts and act outside the harness, measured against each deployment.
• The reviewer-model arm of E20 across model families, and real claude -p sessions with planted faults.
• A fresh blind policy pool for the next release, since E21’s pool has been read.
• Attribution on frontier models with larger samples and partial causes, and cheaper replay through decision-level tests.
• Capture of services (L5) and reconciliation of kernel-sensor effects against calls, not only joining them in analysis.
• External review of the threat models and the format specification, and independent verifiers in other languages written from the specification alone.

10 Conclusion

An agent’s own log is the one account that should never stand alone. Tracekit makes the agent ask a signer it cannot reach (outside dev mode) before each action, signs every decision, approval and result, and each disagreement it detects between capture layers, into a transparency log that witnesses can cosign, and hands anyone a bundle they can verify offline against keys they chose. Its guarantees are stated per deployment and measured by experiments that keep their misses: the log alone cannot see a cut-off tail, policy rules stop half of what they were not written for, and text checks are only as good as the wording they were tested on. Within those limits, it answers the three questions an incident review asks (what did the agent do, was it asked to, and did it say what it was doing) on a record a third party can trust, and adds a fourth, with an interval: which input made it act.

Acknowledgements

We thank the maintainers of the open-source agent projects used in section 8 and the authors of the C2SP, Certificate Transparency and Sigstore specifications on which the evidence format is built. Any errors are ours.

References

1. C. Adams, P. Cain, D. Pinkas, and R. Zuccherato. Internet X.509 public key infrastructure time-stamp protocol (TSP). RFC 3161, 2001. URL https://www.rfc-editor.org/rfc/rfc3161.

2. Anthropic. Claude Code documentation: Hooks reference and monitoring. https://code.claude.com/docs/en/hooks, https://code.claude.com/docs/en/monitoring-usage, 2026. Accessed October 2026.

3. B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025.

4. A. Bhatt et al. Ctrl-Z: Controlling AI agents via resampling. arXiv preprint arXiv:2504.10374, 2025.

5. C2SP. tlog-checkpoint: Transparency log checkpoints. https://c2sp.org/tlog-checkpoint, 2026. Accessed October 2026.

6. C2SP. tlog-cosignature: Witness cosignatures for transparency logs. https://c2sp.org/tlog-cosignature, 2026. Accessed October 2026.

7. C2SP. Signed note. https://c2sp.org/signed-note, 2026. Accessed October 2026.

8. C2SP. tlog-tiles: Tiled transparency logs. https://c2sp.org/tlog-tiles, 2026. Accessed October 2026.

9. C2SP. tlog-witness: Transparency log witness protocol. https://c2sp.org/tlog-witness, 2026. Accessed October 2026.

10. Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, et al. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025.

11. S. A. Crosby and D. S. Wallach. Efficient data structures for tamper-evident logging. In Proceedings of the 18th USENIX Security Symposium, 2009.

12. E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. arXiv:2406.13352.

13. E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tramèr. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813, 2025.

14. M. W. Fagerland, S. Lydersen, and P. Laake. Recommended tests and confidence intervals for paired binomial proportions. Statistics in Medicine, 33(16):2850–2875, 2014.

15. B. Ghosh. Tracekit: Tamper-evident intent–reasoning–action auditing for autonomous coding agents. arXiv:2609.35659v1, 2026. Version 1 of this paper, 28 September 2026.

16. R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger. AI control: Improving safety despite intentional subversion. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. arXiv:2312.06942.

17. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023. arXiv:2302.12173.

18. S. Haber and W. S. Stornetta. How to time-stamp a digital document. Journal of Cryptology, 3(2):99–111, 1991.

19. L. Hinds. What really happened in there? A tamper-evident audit trail for AI agents. nono blog, https://nono.sh/blog/secure-agent-audit, 2026. 22 April 2026.

20. H. Inan et al. Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674, 2023.

21. Invariant Labs. Invariant Guardrails. https://github.com/invariantlabs-ai/invariant, 2025. Accessed October 2026.

22. S. Josefsson and I. Liusvaara. Edwards-curve digital signature algorithm (EdDSA). RFC 8032, 2017. URL https://www.rfc-editor.org/rfc/rfc8032.

23. T. Korbak et al. Chain of thought monitorability: A new and fragile opportunity for AI safety. arXiv preprint arXiv:2507.11473, 2025.

24. B. Kuan. halo-record: Open, tamper-evident runtime records for AI agents. https://github.com/bkuan001/halo-record, 2026. Accessed October 2026.

25. J. Kutasov et al. SHADE-Arena: Evaluating sabotage and monitoring in LLM agents. arXiv preprint arXiv:2506.15740, 2025.

26. Langfuse. Trace Claude Code with Langfuse. https://langfuse.com/integrations/developer-tools/claude-code, 2026. Accessed October 2026.

27. T. Lanham et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023.

28. B. Laurie, A. Langley, and E. Kasper. Certificate transparency. RFC 6962, 2013. URL https://www.rfc-editor.org/rfc/rfc6962.

29. B. Laurie, E. Messeri, and R. Stradling. Certificate transparency version 2.0. RFC 9162, 2021. URL https://www.rfc-editor.org/rfc/rfc9162.

30. D. Ma and G. Tsudik. A new approach to secure logging. ACM Transactions on Storage, 5(1):2:1–2:21, 2009. doi: 10.1145/1502777. 1502779.

31. Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2): 153–157, 1947.

32. National Institute of Standards and Technology. Stateless hash-based digital signature standard. FIPS 205, 2024. URL https://csrc.nist.gov/pubs/fips/205/final.

33. R. G. Newcombe. Interval estimation for the difference between independent proportions: Comparison of eleven methods. Statistics in Medicine, 17(8):873–890, 1998.

34. Z. Newman, J. S. Meyers, and S. Torres-Arias. Sigstore: Software signing for everybody. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2022. doi: 10.1145/3548606.3560596.

35. OpenTelemetry Authors. Semantic conventions for generative AI systems. https://opentelemetry.io/docs/specs/semconv/gen-ai/, 2026. Accessed October 2026.

36. J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009.

37. T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen. NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of EMNLP 2023: System Demonstrations, 2023. arXiv:2310.10501.

38. Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox. In International Conference on Learning Representations (ICLR), 2024. arXiv:2309.15817.

39. A. Rundgren, B. Jordan, and S. Erdtman. JSON canonicalization scheme (JCS). RFC 8785, 2020. URL https://www.rfc-editor.org/rfc/rfc8785.

40. B. Schneier and J. Kelsey. Secure audit logs to support computer forensics. ACM Transactions on Information and System Security, 2(2):159–176, 1999.

41. Sigstore. Rekor v2: a tile-backed transparency log. https://github.com/sigstore/rekor-tiles, 2025.

42. T. Tango. Equivalence test and confidence interval for the difference in proportions for the paired-sample design. Statistics in Medicine, 17(8):891–908, 1998.

43. M. Turpin, J. Michael, E. Perez, and S. R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.04388.

44. T. Yuan et al. R-Judge: Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. arXiv:2401.10019.

45. Q. Zhan, Z. Liang, Z. Ying, and D. Kang. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. arXiv:2403.02691.

Appendix A Reproducing the results

Everything in this paper is in the public repository at the 1.4.1 release. Every number in this paper’s evaluation comes from the committed results files; the project’s technical report regenerates its tables from them, and a test fails when the documentation and the results disagree.

pip install tracekit-ai                     # the package; "tracekit" on PyPI is an unrelated project
uvx --from tracekit-ai tracekit demo --server   # allow, deny, approve; then a tampered copy fails verification
make install && make eval                   # E1-E4 (v1 and v2), E6, E10-E12, E14 --quick, E16-E21; E9 --quick on Linux
python3 eval/e9_signer_perf.py              # E9 full run (gated on Linux); --storage postgres --dsn DSN
python3 eval/e15_durability.py              # E15, POSIX
sudo /opt/tracekit/bin/python eval/e8_insider_v2.py --require-harness   # E8 v2, Linux system mode
python3 eval/e22_verifier_agreement.py      # E22: needs node, the TypeScript build, Playwright with Chrome
make eval-agents                            # E5: real Claude Code runs (spends model usage)
E20_MODEL=1 ANTHROPIC_API_KEY=... python3 eval/e20_alignment.py   # E20 reviewer-model arm (not yet run)
python -m eval.why_bench                    # attribution benchmark, simulated model (seconds)
python -m eval.why_bench --model openai --openai-model qwen2.5:7b --runs 3 --n 10 --n-max 40 --oracle 12 --yes  # local model (Table: attribution)
python3 report/tables.py                    # regenerate the report's tables from eval/results/

Adding Tracekit to an agent.

import tracekit; tracekit.instrument()   # first line of the agent: gates and records every tool and model call
# ... run the agent as usual ...
# $ tracekit last      -> exports the latest run, pins the dev key, verifies it
# $ tracekit view --dev

A policy that extends the server pack.

version: '2026.10-acme-1'
extends: packs/server.yaml
unknown_tools: ask                  # an unmapped tool waits for a person
tools:                              # tool name or glob -> class
  send_email: email
  refund_order: payment
  close_account: app
untrusted: [read_inbox, get_ticket] # results of these tools are untrusted content
deny:
  - id: ACME-D001
    tool: close_account
    pattern: '.*'
    reason: accounts are closed by a person
    remedy: Ask a person to close the account.
ask:
  - id: ACME-A001
    class: email
    field: recipients
    pattern: '@'
    from: untrusted                 # only when an address came from untrusted content alone
    reason: recipient taken from untrusted content
    remedy: Confirm the recipient with the customer first.

tracekit policy lint checks a policy before the signer loads it, and tracekit policy compile prints its canonical JSON and the policy_hash that every decision records.

Appendix B Sample verifier output

The output of tracekit demo --server on Tracekit 1.4.1 (Linux, this paper’s build machine). A scripted agent makes three calls; the run is exported once final and cosigned by an in-process test witness, verified, then a copy with the denied call rewritten as allowed is verified again.

== agent run (SCRIPTED: fixed tool calls, not a model)
  Bash 'ls': allow
  Bash 'sudo rm -rf /': deny TK-D001 TK-D004 TK-F003
  Bash 'echo "unterminated': ask TK-SHELL-PARSE
    approved apr-5be74941c1e60f28e8b399ea9353f984 (self-approved: True)

== verify offline with the pinned trust config
[PASS] manifest - every file is listed with its SHA-256 and nothing else is in the bundle
[PASS] checkpoint - tracekit.local/6fb236bd... at tree size 12, signed by its pinned log key
[PASS] witness quorum - 1 pinned cosignature(s), 1 required
[PASS] keys - 1 record key(s) from 1 key record(s)
[PASS] signatures - 11 record(s), keys valid at their position
[PASS] run chain - run 'ca0529be...' of tenant 'default', contiguous from run_seq 0
[PASS] inclusion - first and last records of 1 run(s) are in the checkpointed tree of size 12
[PASS] coverage - layers L2 (L3 absent); 0 call(s) reconciled; unreconciled: none
[PASS] coverage matrix - model calls not covered, tool calls reported (adapter), network not covered,
       processes not covered, services not covered
[PASS] tiers - 3 tool call(s): T1 0, T2 0, T3 0, untiered 3
[PASS] args source - 3 record(s): raw 0, parsed 3, coerced 0
[PASS] isolation - signer-reported: same-user 1 run(s)
[PASS] fail-open classes - none
[PASS] key assurance - asserted: the log declares its record keys; none is attested

Integrity: VERIFIED.
Assurance: dev; records ed25519; checkpoint Ed25519 only; cosigned ed25519 by witness.demo.tracekit.local
(customer); approvals: self; key retirements not proven complete.

== tamper test: rewrite the denied call as allowed in a copy, then verify again
[FAIL] manifest - files differ from the manifest: runs/f5906de3....jsonl (changed)
[FAIL] signatures - 11 record(s), keys valid at their position
        seq 4: hash does not match the event
...
Integrity: FAILED.
  original bundle exit 0, tampered bundle exit 1

Note what the report says about itself: the coverage matrix admits that only tool calls were captured (no gateway, no network or process layer); isolation is same-user; and assurance stays dev although a witness cosigned, because the held call was approved by the run’s own identity.

Appendix C Finding catalogue

Table 34: Findings of tracekit analyze --v2, with the reasons a call can be not checkable.

analyze --v2 also runs version 1’s say-versus-do checks against the signed records: claimed test, build or lint runs, commits, pushes or installs that never ran (TK-X101–X104, high), denials of actions that did run (TK-X111–X114, critical), and a hidden force push (TK-X120, high). There is no TK-X209.

Appendix D Glossary

Cite

@misc{ghosh2026tracekit,
  author        = {Bravish Ghosh},
  title         = {Tracekit: Tamper-Evident Intent-Reasoning-Action Auditing for AI Agents},
  year          = {2026},
  eprint        = {2609.35659},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2609.35659},
  note          = {Code: \url{https://github.com/Cygnux-Labs/Tracekit}}
}