The internal architecture of Tracekit, explained part by part: where the signer sits, how one tool call is decided, how records are signed and chained, and how someone who trusts neither the agent nor its operator can check a run offline.
This article describes Tracekit as it is on main on 2 November 2026, after release 1.5.2.
One v2 record, one line of the log (illustrative values; JCS is RFC 8785 JSON canonicalisation):
{"v": 2,
"event": {"type": "policy.decision", "log_id": "9f3c…", "tenant": "acme",
"run_id": "run-7a2…",
"seq": 1041, "prev_hash": "sha256:4be1…",
"run_seq": 12, "run_prev_hash": "sha256:c09d…",
"tool_call_id": "toolu_01…", "tier": "T2",
"args_commitment": "hmac-sha256:…",
"data": {"decision": "ask", "rule_ids": ["TK-P001"],
"policy_hash": "sha256:77a0…"} …},
"hash": "sha256:SHA-256(JCS(event))",
"alg": "ed25519",
"kid": "sha256:SHA-256(DER public key)",
"sig": "Ed25519 over JCS({alg, kid, log_id, seq, prev_hash, tenant, run_id,
run_seq, run_prev_hash, hash, t})"}
- Numbers the agent never chooses.
seqandrun_seqcome from one writer thread inside the signer. - Two chains at once. Every record points back to the log's previous record and to its run's previous record.
- Arguments as commitments. The record proves what the call carried without carrying it, so a bundle can be shared.
1 Why the architecture decides what a log is worth
When an AI agent does something it shouldn't, three questions follow. What did it do? Was it allowed? Can we trust the record? Most agent logs can answer the first two. They cannot answer the third, because the agent, or code running as the agent, can edit them.
Agents now run shell commands, change files, query production databases, send email and move money. The record of what they did is how a team, a customer or an auditor finds out what happened. A record written by the party being checked, and stored where that party can reach it, proves very little.
Tracekit is an open-source flight recorder for AI agents (github.com/Cygnux-Labs/Tracekit). It checks each tool call routed through it before it runs, signs each call and decision into a hash chain, and exports one .tkb file per run that anyone can verify offline. This article describes how it does that. Each part names the file or function that implements it, so every claim here can be checked against the source.
We think a trustworthy record of agent activity needs four properties:
- Out of reach. The agent never holds a signing key and never assigns a sequence number. It can only ask.
- Decided before it runs. A call that matters is checked against a policy the agent can't change, before the action happens.
- Says what is missing. When a layer saw something that didn't reach the recorder, a signed record says so.
- Checkable by strangers. A verifier needs no network and no trust in the signer's operator. It brings its own trust config.
We know of no logging library that gives all four. They have to come from the system around the agent. Tracekit gets them by following the design of Certificate Transparency: signed records in an append-only log, summarised by a Merkle tree whose checkpoints independent witnesses cosign. The agent-specific parts sit on top of that base: deciding tool calls, holding them for approval, reconciling what the model asked for with what was recorded, and signing gaps.
Figure 1 shows the whole system. Everything that can act sits on the left and holds no key. The signer in the middle owns the keys, the sequence numbers and the policy. Its records leave in two ways: as checkpoints that witnesses cosign, and as bundles that verifiers on the right check without trusting the signer's operator.
Figure 1: The system map. The bus on the left is the only way in: every capture path ends in a signer RPC. Reconcile feeds its gaps back into the same writer, so they are signed and chained like any other record. Code: tracekit/signer/service.py, pipeline.py, reconcile.py, tracekit/verify/v2.py.
The rest of the article follows a tool call through this picture from left to right. Section 2 covers the boundary between the agent and the signer, since the value of every record depends on it. Section 3 covers how calls reach the signer. Section 4 covers how a call is decided. Sections 5 to 7 cover how decisions become records, how the log resists rewriting, and how missing activity is reported. Section 8 covers verification, Section 9 the tools that read verified runs, and Sections 10 to 12 the measurements, the design decisions and the known limits.
2 The trust boundary
A record is worth what it would take to rewrite it. So the first design question is who could rewrite it, and that depends on where the signer runs relative to the agent.
2.1 Where the signer runs, and how isolation is measured
Tracekit runs the same signer in three kinds of place. The agent's code is identical in every mode: an environment variable, TRACEKIT_SIGNER, points it at the signer. What changes is whether the agent's operating-system user can reach the signer's files.
Figure 2: The arrow is always the same signer RPC. What changes is whether the agent's user can reach the signer's files. Code: service._isolation, _check_harness, harness_helper.py; docs: concepts/deployment.md.
Problem. An agent that can edit the log it will be judged by makes that log worthless. Signatures don't help if the agent can read the key that makes them.
How it works.
- Dev signer. It starts by itself as the agent's user, one per home directory. It records
isolation: same-user, counts its own user as an analyzer so any findings read as self-asserted, keeps content for 7 days and exits after 15 idle minutes. On Windows it serves loopback TCP instead of a Unix socket. - Linux system mode (
tracekit init --v2 --user AGENT). The signer runs as its own OS user in a hardened systemd unit: no capabilities,ProtectSystem=strict, a syscall filter. A root-owned/etc/tracekit/client.jsonstops the agent pointing its hooks at a decoy signer.tracekit doctorflags an agent user that is root or in the sudo or docker groups. - Sidecar, Compose or central. Another user, or another host over HTTPS. A central signer has one writer per replica, and runs are routed to the right replica by an id prefix.
- Isolation is measured, never claimed.
_isolationreads the transport identity. The signer's own uid issame-user. Another uid isseparate-user. Kubernetes service accounts, mTLS, OIDC and non-dev tokens areremote. Anything else isunknown. The result goes intorun.registered, and no request field can set it. - Harness binding (Linux, opt-in).
register_runwalks the caller's process chain up to a registered harness, such as Claude Code, and records that process's pid and start time as attested. Later calls from the run's owner must come from below that process. The signer itself holds no ptrace capability. A small root helper holding onlyCAP_SYS_PTRACEandCAP_DAC_READ_SEARCHanswers it, and nobody else.
Why. Dev mode protects nothing against the agent itself; it exists so people can try Tracekit in a minute. Measuring isolation means the reader learns how strong the boundary was from the signer, not from the party being checked. Isolation and assurance are kept separate on purpose: isolation stops the agent rewriting the log, and only an independent witness lets you catch the operator doing it. A well-isolated signer with no witness still verifies at assurance level dev, the lowest, because no witness or anchor in Rekor (Sigstore's public transparency log) that the verifier trusts vouches for its checkpoint.
Limits. Harness binding doesn't stop a process the agent starts inside the harness session, or the agent's user starting its own copy of the harness. macOS has no per-message credentials, so a caller's identity there is fixed when the socket connects. Without harness binding, the agent's uid can read its own run tokens.
| Where the signer runs | Who can reach the keys and log | Recorded isolation |
|---|---|---|
| Laptop dev | the agent's own user | same-user |
| Laptop system mode | root and the signer's OS user (Linux; macOS experimental) | separate-user |
| Container or Kubernetes sidecar | the signer's uid and the node's administrator | separate-user, or remote over HTTPS |
| Docker Compose stack | the Docker host's administrators | separate-user |
| Central signer over HTTPS | the signer host's administrators | remote |
2.2 Caller identity, run tokens and grants
Everything the signer decides depends on who is asking. If a caller could say who it is, the agent could claim to be an approver, an operator or a gateway. So the rule is simple: the transport's authenticator establishes the caller's identity, never the frame's content. On Linux it does so again for every frame, and over HTTPS for every request.
- Unix socket. The uid comes from the kernel's peer credentials at connect time and, on Linux, from
SCM_CREDENTIALSon every frame. A frame whose uid differs from the connecting one is refused. A process whose real and effective uids differ is refused too, unless it attachesSCM_CREDENTIALScarrying its effective uid itself. A socket handed to another user's process therefore can't speak with the connecting user's identity. - TCP (Windows dev). A mutual HMAC challenge: each side proves it holds the token without sending it, so a process squatting on the port learns nothing it can replay. The endpoint file is written
0600withO_EXCL|O_NOFOLLOW, and the client acceptstcp://only on loopback, because frames after the handshake are plain text. - HTTPS. Every request is authenticated again, with no sessions, over TLS 1.2 or later. Failed attempts are rate-limited per address and in total.
- Run tokens.
HMAC-SHA256(run_token.key, tenant, run, identity). A child run must present its parent's token. Tokens are HMACs rather than signatures, so a token can never be mistaken for, or used to obtain, a signature over evidence. - Grants. Default grants apply only to an unconfigured local uid or the dev token. Every other identity gets exactly its
authorizeentries. Content reads, decision imports, on-behalf answers, approval review, tailer heartbeats, effect observations and operator calls are never default. An identity can't be both an operator and a run owner: startup refuses that config, and a runtime check refuses the call. This stops an agent resuming its own quarantined run.
Code: transport/unix.py, tcp_dev.py, http.py · identity/uid.py, token.py · signer/runtoken.py · service._grant, DEFAULT_GRANTS
2.3 One key per purpose
The signer keeps a separate key for each job, so a mistake with one key can never produce another key's signature. Each key file is created 0600 with O_EXCL, fsynced and linked into place inside a 0700 directory. A key file of the wrong size stops startup, so a key is never used cut short. Each key is mlocked in memory, best effort and only Python's copy of it, not the copies the crypto library keeps. Before any key is read, the process turns off core dumps and, on Linux, makes itself non-dumpable. The result goes into hygiene.json, which tracekit doctor checks.
| Key | Used for |
|---|---|
record.key | Ed25519 signatures on records, unless an issuer certifies short-lived record keys instead |
log.key | Checkpoint notes only, stable for the log's whole life. tracekit doctor checks that it differs from the record key (D-KEYS-DISTINCT). Can live in AWS KMS |
run_token.key | The HMAC behind run tokens |
args_salt.key | Per-record salts for argument and output commitments |
approval_args.key | AES-GCM for the sealed copy of an approval's arguments |
content.key | Wraps each run's content-vault key; created only when the vault is on |
registry_salt.key | Per-tenant salts, so run-set registries never carry tenant names |
snapshot.key | An HMAC over state snapshots; a snapshot that fails it forces a full replay |
rekor.key | The P-256 key that publishes anchors to Rekor |
Key changes are records. signer.epoch declares a key, and key.retire, carrying the last seq the key signed, ends it. Validity is positional, so a retired key's later signatures don't verify. With an issuer configured, the signer makes a fresh in-memory record key at each start and again at two-thirds of the certificate's life; the certificate is witnessed before the signer gets it, and writes are refused once it expires. A KMS log key is checked against its public key on every note, and a log-key outage longer than 5 minutes writes a degraded_unanchored gap.
An optional second signature line on each checkpoint, made with SLH-DSA, hedges against a future break of Ed25519. We chose a stateless scheme because restoring a snapshot of a stateful one (XMSS, LMS) would reuse one-time state. The line is never sent to witnesses or Rekor, and costs about a second per note.
Code: service._secret · signer/logkey.py (harden, AwsKmsKey) · format/slh_dsa.py · docs/issuer.md
3 Capture tiers and custody
A hook reports what the agent says it did. That is useful, but it lives inside the agent's process, where it can be skipped, patched or simply not called. For the actions that matter, Tracekit can instead put the action itself on the far side of the boundary, so that it cannot happen without the signer seeing it. Every record says which kind of path its call took.
Figure 3: On a T1 path the agent holds the credential and reports its own calls. On a T2 path the credential exists only in the executor or gateway, which asks the signer before it acts. Code: service._as_caller, _hold · executor.py · gateway.py · mcp_gateway.py · a2a_gateway.py.
| Tier | Path | What it proves |
|---|---|---|
| T1 | Hooks, framework adapters and the SDK, inside the agent's process | What the agent reported |
| T2 | The tool executor, the MCP and LLM gateways, and the A2A gateway (experimental) | What crossed a boundary the agent doesn't control |
| T3 | Imports and decision imports (for example OpenTelemetry) | Only that the signer received them |
Problem. An agent that holds a credential can use it without asking. A hook in its own process can be skipped, patched or not called at all.
How it works. The credential lives only in the executor or gateway. The agent asks the gateway, the gateway asks the signer, and only then does the call run. _hold refuses a decide, approve or complete of a gateway-held call from anyone but that gateway, so the agent can't complete, consume or abandon the executor's calls.
When a T2 call needed approval and its ask rule sets approval: {executor: t2}, the executor receives the signer's sealed copy of the arguments when it consumes the approval, so what runs is byte for byte what was approved. The executor and the MCP and A2A gateways fail closed whatever the run's fail modes say; the LLM gateway follows the run's model class. Gateway records spend the gateway's own quota, so the agent can't starve them.
T2 records carry source: gateway and tier: T2. The verifier counts each call at the weakest tier its records claim (untiered, then T3, T1, T2) and warns when a call's decision and result came by different paths. run.final carries a coverage matrix per action class: enforced, observed, reported or none.
Why. Custody moves the action out of the agent's process. An action whose credential exists only in the executor can't happen unseen, whatever the agent's code does.
Limits. It covers only actions routed through a gateway. An agent that also holds the credential can act around it. "Enforced" in the coverage matrix is the operator's posture claim, and the record marks it as asserted, not attested.
4 Deciding one tool call
The agent can change anything inside its own process, including a policy check. So the decision has to be made by the signer, over the raw arguments, with the signer's own policy and its own view of what kind of tool is being called. The agent asks before it acts. The signer authorizes the caller, digests the arguments, checks where they came from, and decides the call under every reading of its name. After that, detection, live constraints and any hold on the run can only make the answer stricter.
There are four verdicts: allow, flag (answered as allow, recorded as flagged), ask (waits for an approval) and deny (never runs).
Figure 4: One tool call, from the agent's question to its signed outcome. The top row runs for every call; the verdict then picks one of the lanes below. Code: service._decide, _evaluate, _stricter, _complete, approval_consume.
4.1 The decision path, step by step
- The arguments are parsed strictly and digested as
sha256(JCS({tool, args})). Arguments that aren't canonical JSON are denied withTK-ARGS-INVALID. - Provenance and policy matching run off the single writer thread, so a slow match never stalls other runs.
_evaluatedecides every reading of the call's name (Section 4.2)._stricterkeeps the stricter result (allow, flag, ask, deny in rising order), the union of rule ids, and any oversize or nondeterministic mark.- An exception inside the engine is a deny,
TK-ENGINE-ERROR. Only the exception type is logged, because its message could carry argument text. - A deny or ask is remembered per call id and attempt. A retry of the same attempt with the same digest gets the same answer, unless it was a hold that has since ended. Allows are not remembered, so a constraint or hold added since can still tighten the next answer.
- Every denied digest is kept. A later allow for the same digest becomes a signed
decision_flipgap. - After the policy, only tightening steps run: live constraints (
TK-CONSTRAINT), a sum limit that can't read its amount (TK-LIMIT-SUM), and a hold or quarantine on the run. - The signed decision carries a fresh
decision_id, the policy hash, the engine id, the rule ids, a nonce, the arguments commitment andexpires_at(300 seconds). The rules' reasons and remedies go only into the response to the client. The record holdsreasonsonly forTK-OVERSIZE, a regex timeout orTK-ENGINE-ERROR.
After the verdict, complete must quote the decision_id and arguments that open the decision's commitment, or it is refused as args_mismatch. A call that ran after a deny, or after an ask with no consumed approval, becomes capture.gap{executed_against_policy}. An allowed call that never completes becomes decided_not_completed at run.final. For a tool whose results are untrusted, such as a web fetch, decide answers send_result: true, so the client sends the result itself rather than its digest, and provenance can index it.
4.2 Every reading of a name
A rule written for Bash is useless if the same call can arrive as bash, or as mcp:shell/Bash. Frameworks rewrite names (CrewAI sends bash for Bash), and an agent under prompt injection may try aliases deliberately. So the signer decides the call under every name it could reasonably mean, and keeps the strictest answer.
Figure 5: The readings are: the name as sent; the listed name it spells with case, accents and punctuation ignored; for mcp:server/tool, the bare tool name; and, if the client's class hint disagrees with the policy's class, the hinted class. A spelling that two listed names share is not used. The three name readings merge through _stricter: the higher verdict, with the rule ids of all three. The class hint's reading doesn't merge: the stricter of the two decisions is signed with its own rule ids alone, and the rule ids of both go to detection (data.detect).
Deciding every reading means a renamed spelling of a listed tool can't dodge its rules. A client's hint can only tighten, because the reading under the class the policy maps the tool to always counts, and a disagreement is itself signed. Detection sees the rule ids from both readings, so a mismatch can't hide a step of a sequence.
4.3 The policy engine
A regular expression over the raw text of a command is easy to fool and risky to run. Quoting, wrappers and nested shells hide the real command. A commit message that mentions rm -rf trips it. A hostile input can make a backtracking regex take minutes. The engine in tracekit/policy2/ is built to avoid each of those.
Policy comes in packs that layer through extends. A child's rule with the same id replaces the parent's; tool maps merge and lists union. Inside the engine, a call becomes a class, a subject and, for the shell, a parsed argv. The highest section that matches wins. Anything the engine can't read cleanly becomes a rule id of its own, never an allow.
Figure 6: Left: the shipped packs. Right: how one call is matched. The dashed exits are deliberate: a call the engine can't read cleanly gets a rule id, and its pack decides how strict that id is. unknown_tools and on_parse_error are themselves pack settings. Code: policy2/compile.py, engine.py, shell.py.
Compile. Packs are read by a built-in YAML subset rather than PyYAML, which would keep the last of two duplicate keys and expand anchors. Lint reports every error at once and refuses unknown keys, a field or class a rule can never see, reserved ids, and an unless that matches everything. The compiled policy is hashed (policy_hash), and every decision names that hash.
One pattern language. Patterns are limited to what RE2 and the fallback regex module read the same way. Lookarounds, inline flags and an unescaped { are refused; $ means end of text; nested repeats are capped at 1,000; (a+)+ shapes are refused. A decision therefore doesn't depend on which engine made it.
Shell parsing. shell.py produces a record for every simple command, including those reached through wrappers, runners, sh -c, eval, command substitution, heredocs, ssh, find -exec, trap and interpreter one-liners. argv[0] is reduced to its basename, case-folded, with .exe dropped. Commit messages and grep patterns are kept out of the targets, so git push && ls -f isn't read as a force push. A command whose program can't be read, such as a shell fed from stdin, is marked opaque. Work is bounded at depth 32 and 256 Ki characters.
When reading fails. Unparseable or opaque shell gets TK-SHELL-PARSE in the pack's on_parse_error section. SQL that no dialect can read gets TK-SQL-PARSE, which always asks. A subject over 64 KiB gets TK-OVERSIZE, and a shell command that size is denied; oversize subjects are never cut, because cutting would let content hide past the cut. On the fallback engine all matches of one decision share one second; a timeout counts the rule as a deny and marks the decision nondeterministic, because RE2 never times out and the two engines could otherwise disagree.
Limits. Shell variables aren't expanded and encodings aren't decoded. An unknown wrapper, such as bwrap, hides its command. Interpreter code is only scanned for literals. The policy hash is over sorted compact JSON, not yet JCS.
On main the packs hold 107 unique rule ids, grouped in families:
| Family | What it covers | Ids |
|---|---|---|
TK-D | Coding denies: destructive and escape commands | 15 |
TK-A | Coding asks: deletes, merges, secrets, money, persistence. 37 are in the coding pack; TK-A032 is in the dev pack only, so server and strict deployments have 37 | 38 |
TK-F | Coding flags: recorded, not held | 16 |
TK-S | Server rules for every tool | 7 |
TK-B | Browser | 6 |
TK-DB | SQL | 5 |
TK-M | Money and mass email | 5 |
TK-C | Cloud identity and deletes | 4 |
TK-N | HTTP egress | 3 |
TK-P | Acting on untrusted content | 3 |
TK-L, TK-Q | Detection limits and sequences | 3 |
TK-DEMO | Demo rules in dev | 2 |
Some ids belong to no pack because they name a condition rather than a pattern: TK-UNKNOWN-TOOL, TK-SHELL-PARSE, TK-SQL-PARSE, TK-OVERSIZE, TK-ENGINE-ERROR, TK-ARGS-INVALID, TK-CONSTRAINT, TK-LIMIT-SUM, TK-HOLD, TK-QUARANTINE, TK-APPROVED and the TK-APPROVAL-* family.
Each class reads specific argument fields, and rules match subjects derived from them:
| Class | Argument fields | Subjects rules can match |
|---|---|---|
shell | command, cmd, commands | argv of each command; effects (write and delete targets) |
fs | file_path, notebook_path, path | path, operation, digest of the content |
http, browser | url and the request body | host and whether it is internal (as a browser reads it), sends_to |
sql | sql, query, statement, database | code per dialect with comments and literals blanked, verb, database |
payment | amount, amount_cents, currency, payee keys | amount, currency, payees |
email | to, cc, bcc, recipients, attachments | recipients, attachments |
mcp, a2a | the tool or method name | server and tool, or the A2A fields |
Every class also has target. A name that isn't in tools: has class unknown, which has no fields for a class rule to read. The server pack therefore makes an unknown tool wait for a person, and the coding pack only flags it.
4.4 Provenance: acting on untrusted content
Prompt injection works because a web page that says "email the list to x@vendor.example" and a task that says the same thing look identical in the resulting tool call. Provenance remembers which values arrived in untrusted content, so that a later call using one of them can be held.
Figure 7: A from: untrusted rule needs both conditions: its pattern must match, and the value must be one the index saw in untrusted content. Code: tracekit/provenance.py, service._provenance; packs server-comms.yaml (TK-P001 recipients, TK-P002 payees), server-net.yaml and browser.yaml (TK-P003 data sent to a host).
What counts. Results of tools whose class or name is in the pack's untrusted list. The coding pack lists http, mcp, WebFetch and WebSearch. A pack's list is the union of its own and those of the packs it extends, so the server pack's list is http, browser, mcp, WebFetch and WebSearch. Content observed with trust: untrusted counts too. Results of other tools, such as SQL or the shell, are not indexed either way: a tool can echo what the agent put in its own arguments, and indexing that output as trusted would launder an injected value. When the same value was seen as trusted, trusted wins.
Fails closed. If the index isn't complete, every lookup answers untrusted. The index is incomplete when a result arrived as a digest instead of content, a client_seq gap hides events, a result had too many values, the run's index is full, the signer restarted, or reading the values took more than a second. In each case the index can no longer say a value is not untrusted.
Why HMAC keys. Index keys are HMACs under a per-process key that never touches disk, so the signer never keeps brute-forceable hashes of card numbers and addresses at rest. The decision records which field matched, what kind of value it was and where it came from, never the value itself.
Limits. Matching is verbatim, so a paraphrased, split or encoded value passes. A curl inside a shell command isn't indexed. trust on an observation is the caller's claim. This is a tripwire against injected content, not a defence against a malicious agent program, and not information-flow control of the kind CaMeL or FIDES provide.
4.5 Approvals
An ask turns into a question for someone with the right to answer it. The design goal is narrow: an approval approves one call, with exactly those arguments, once, from someone allowed to give it.
Figure 8: From an ask to a consumed approval. Code: service._answer, _eligibility, _capped, _passkey, approval_consume, _reusable · identity/webauthn.py · slack_approvals.py.
Who may answer. Only listed approvers, or break-glass with a written reason (never through Slack). Without an approvals section in the config, anyone in the tenant may answer, and self-answers are recorded as such. With an approvals section, there is no self-approval unless the config allows it, and then the record labels it. A persons map links one person's identities, so someone can't approve from Slack what their own agent asked for. The run's principal never answers, and that check runs before any self-approval or break-glass rule.
Agent approvers. Some teams want a second model to review low-risk asks. Under prompt injection, the requesting agent controls every argument the reviewing agent sees, so this path is fenced tightly. Each agent approver must be one exact identity with no human role. A call is eligible only if every rule that held it allows an agent, the tool is named exactly in tools:, the run is not a replay, the decision was deterministic and provenance is clean. Arguments must fit closed bounds (exact JSON types, 16 KiB at most). Budgets are rebuilt from the records: 10 reviews per run and 1,000 per tenant per hour. Holds, constraints and limit asks always go to people. Shadow mode only writes approval.suggested. An agent's approval can open a veto window (veto_s, off by default) before it can be used, and a kill switch revokes and escalates.
Passkeys. A rule with passkey: required needs a WebAuthn assertion whose challenge is the hash of the approval id, the binding digest and the decision. The signer checks origin, relying party, user presence and verification, and that the signature counter grows. Because the challenge binds every field, an assertion can't approve another approval, other arguments or another run.
Consume. An approved call still has to be consumed, just before it runs. approval_consume refuses, in this order, when:
- the run is quarantined;
- the run is held, unless this approval belongs to the holding incident;
- the client's hint names a different approval;
- the arguments aren't strict JSON;
- there is no approval (an allowed call passes; any other gets
TK-APPROVAL-REQUIRED); - the arguments don't open the approved commitment, which also writes
approval.binding_mismatch; - the approval isn't in the approved state;
- it has expired or passed its use-by time;
- it is still inside its veto window.
Only then is it consumed, with a signed approval.consumed, and TK-APPROVED returned. Consume is separate from approve so that the actual arguments are checked one last time, right before they run, and so that one approval runs exactly one call. complete counts an ask as approved only once it was consumed, and detection counts it only then too.
Approve, then retry. No coding agent shows a hook's output while the hook waits, so the v2 hook blocks at once and prints the approve command. When the agent retries under a new tool call id, the earlier approval is reused only if run, tool, path, policy hash and argument digest all match, its rule ids cover the new decision's, it is approved and unexpired, and the latest answer wasn't a no. Nothing the agent sends can name an approval into use; its hint only picks among approvals that already qualify.
4.6 Constraints from the person's own words
A person often says what not to touch in the prompt itself: "don't change the migrations". The agent can forget or ignore that, and the policy doesn't know about it. The hook reads the typed prompt with a fixed grammar. constraint_add signs the kind of constraint and a salted commitment to the lower-cased file name, never the name itself. From then on, any non-deny decision that would write a constrained file becomes an ask, TK-CONSTRAINT.
Constraints only tighten, so anyone may add one: a made-up constraint only makes the agent wait. Past 32 file constraints in a run, the run fails closed and every write is held. The limits are plain: only fixed English phrases are understood, writes the parser can't see (through variables or scripts) aren't held, and matching is by base name, so a same-named file elsewhere also matches.
4.7 Detection and containment
Single calls can each look fine while a run as a whole goes wrong: a string of refunds, or a read of secrets followed by an upload. Detection watches sequences and totals across a run, a session, an owner or a principal. Containment can stop a run while it is still going.
| Detection | Containment |
|---|---|
Sequences of rule ids or labels, and limits by count or by sum. A sum keeps one total per currency and needs a non-negative integer amount and a three-letter currency; anything else makes the call an ask, TK-LIMIT-SUM. | Modes: off, shadow, enforce. Actions: notify, hold, quarantine. Class mismatch, refusal bursts and tamper only notify. |
| Triggers: executed against policy, denied calls, provenance, class mismatch, refusal burst, tamper, unexplained effect. | Hold: live approvals expire and every call becomes an ask, TK-HOLD, which only a person who isn't the owner, principal or requester may answer. With no such person, a hold becomes a quarantine. |
| What counts: a call decided allow or flag, or an ask once its approval is consumed. Denies and unconsumed asks never count. A call of unknown class matches every step, so it fails closed, but one call can't chain steps by itself. | Quarantine: every call is denied, TK-QUARANTINE, and the model loop stops. It is final. |
| Windows are fixed, on the records' own clock. A scope idle for an hour resets. Caps: 100 incidents per run per trigger, 60 per tenant per minute, 20 of them from one run. | New runs of a held owner and principal, and child runs, start held for the cooldown (an hour by default), so dropping the principal gains nothing. Operator calls are never default grants, and are refused to anyone who owns runs. |
Detection is a pure function of the signed records. It reads only rule ids and labels, runs no regex, and is fed each record by Tx.emit as it is written. So the verifier and backtesting can recompute the same incidents later. This answers an operator who claims a run was contained when it wasn't.
Shadow, then enforce. In shadow mode an incident records what it would have done and nothing acts. In enforce mode _respond acts in the same writer step that wrote the triggering record, so no later call can slip in between. If detection state is unavailable while holds are enforced, the run is held under a detector_unavailable incident.
Containment is enforced on T2 paths. A T1 adapter is told to stop, but it runs in the agent's process and could ignore that. Detection state is per signer replica, the verifier recomputes run-scoped incidents only, and fixed windows let a burst reach twice the limit across a window boundary.
Code: signer/detect.py · service._respond, _contain, _contained · docs/incidents.md, concepts/containment.md
4.8 Fail modes and quotas
From the agent's side, two failures look alike: the signer is down, or the signer said no. Treating them the same would let an agent provoke a refusal and then act unrecorded. Separately, one noisy agent shouldn't be able to flood the log or starve other runs.
- The signer's config maps each tool class to open or closed, closed by default. The map is recorded in
run.registered, and the verifier warns on every fail-open class, because calls in it may have run unrecorded. - When the signer can't be reached, the SDK answers from that map for the call's class. A refusal from a live signer never fails open, and a failed consume blocks the call.
- In Linux system mode, hooks fail closed unless the root-owned client config says open, and closed if that config can't be trusted. The executor and the MCP and A2A gateways always fail closed.
- The client counts every event, even when it can't connect, so a fail-open call later shows up as a
client_counter_gap.
This rule has a history. An early insider test showed that an agent could stop the signer and then act unrecorded. Version 0.2.1 fixed it by making system mode fail closed by default (dev mode didn't change), and it is why a refusal is never a reason to fail open.
Quotas are per caller identity: 200 events a second with bursts of 400, 100 open runs, 100 pending approvals, 8 concurrent waits, 64 streams per run, 1 MiB frames and strings. Buckets are deliberately per identity and not per tenant: an app may assert any tenant, and a per-tenant bucket would hand it a fresh burst for every tenant it names. Refusals are counted per identity and code and written every 60 seconds as one signed refusal.summary, so a caller hammering a limit can't make the signer write one record per refusal.
Code: sdk/client.py (fail_open) · client.py (system_fail_closed) · signer/quotas.py
5 Writing the evidence
5.1 The single writer and a run's lifecycle
Order is evidence. If clients assigned sequence numbers, an agent could skip one or reuse one. So every record in the log is written by one thread, and that writer alone assigns sequence numbers and chain heads. The log's seq therefore has no holes. When a client skips one of its own client_seq values, the gap is recorded by the signer, never something the agent arranged.
Figure 9: A run moves through five states, and the signer itself writes the records that close it. The closing window stretches for calls a gateway still holds and for observers that report late. Code: service._finish_run, close_run, delegate_run, pipeline.Tx.
The writer. Request handlers queue work. The writer drains up to 512 items at a time, assigns both chains, signs, and stores them in one batch. Policy matching and checkpoint signing happen off this thread.
Each item runs inside a Tx with an undo log. A refused item rolls back to its mark, so it leaves no state anywhere. A storage failure rolls back the whole batch, and every caller is told to retry.
Idempotency. A request_id is scoped to the caller's identity. The same id with the same payload returns the cached answer; another payload is a conflict. Up to 100,000 are remembered, at most 10,000 per identity, so no caller can evict another's.
Client counters. Each client process has one stream with its own client_seq. A skipped value becomes a signer-written client_counter_gap and marks provenance partial. A reused value is refused.
One writer, enforced. A file lock, or a Postgres advisory lock, makes sure two signers never write one log.
Durability. Ack on write is the default: the file store syncs in the background within 5 ms, and Postgres runs with synchronous_commit off. A power loss can lose the last acknowledged records. On ack on fsync, one client calling decide and complete takes about 7 ms per call at the median, against 0.45 ms on ack on write (E9, a quick macOS run committed on 10 October 2026, before 1.0). Checkpoint notes, anchors and snapshots always commit synchronously.
After a storage failure, the first record the recovered writer writes is a signed signer_unavailable gap with the outage's start and end. A torn last line is set aside and also recorded as a gap. In the disk-full experiment (E14), 41 calls were acknowledged, none were lost, and two gaps were written: the signer_unavailable gap, and a client_counter_gap covering the 22 calls refused during the outage.
On Postgres, the signer's role may only SELECT and INSERT on records, registry, notes and anchors, so the store check finds a row edited behind the signer's back.
Limits. Snapshot and content tables allow updates, which is why snapshots are HMAC'd. A database superuser is detected, not prevented. The tailer heartbeat proves that a process holding the token is alive, not that it is reading the real transcript.
5.2 Record format and the two chains
A signature only checks if the signer and every verifier hash exactly the same bytes, in every language. And a genuine signed record must not be usable anywhere except its own place: not at another position, in another run, or in another log. The record format is built around those two requirements.
Canonical bytes. Events are serialised as RFC 8785 JSON (JCS). A strict parser runs first and rejects bad UTF-8, duplicate keys, NaN and Infinity, integers outside the exact double range or longer than 16 digits, lone surrogates, and nesting deeper than 128. That leaves no input that two languages' JCS implementations could hash differently.
The envelope. A record is one line of JSON with exactly six keys:
{"v": 2, "event": <event>, "hash": "sha256:<hex>", "alg": "ed25519", "kid": "sha256:<hex>", "sig": "<base64>"}
hash = sha256(JCS(event))
kid = sha256(DER SubjectPublicKeyInfo) # so it also fixes the algorithm
sig = Ed25519 over the signature message:
JCS({alg, kid, log_id, seq, prev_hash, tenant, run_id, run_seq, run_prev_hash, hash, "t": "tracekit.record.v2"})
A verifier checks the parts in order: shape, hash, alg on the pinned list, kid known, alg equal to the key's own, then the signature. Ed25519 is checked strictly: S < L, and no small-order or non-canonical keys.
Each field in the signed message has a job. hash binds the event body. The position fields mean a genuine record can't be moved to another place in the log or into another run. log_id lets a verifier reject another log's record. The type tag t separates purposes, so a signature made for one purpose (a record, a checkpoint, a PCM message) never verifies as another; a caller can't supply its own tag.
Figure 10: Every record sits in two hash chains at once: the log's, by seq, and its run's, by run_seq. Only the writer advances either head. Each record hash becomes a Merkle leaf; checkpoints commit to the tree, and witnesses cosign them, so a rewritten history can't match what they already signed. Gaps, incidents and reconcile records go through the same Tx.emit, so no record type is exempt. Code: pipeline.Tx.emit, format/sigmsg.py, format/checkpoint.py, bundle_v2.py.
Why two chains. seq and prev_hash order every record in the log. run_seq and run_prev_hash order one run, and both start from the zero hash. The run chain is what lets a bundle carry a single run: it ships the run's records plus inclusion proofs for its first and last, and the chain covers everything between. The first version of the format had to ship every record from the start of the log, with stubs for other runs that weren't bound to any run. The log chain still matters: key validity is positional on seq, and registry leaves point at a seq. The bundle verifier checks the run chain; the log chain is checked by the signer's store check, fsck and the monitor.
Who writes what. Sources are hook, proxy, transcript, signer, sdk, gateway, migrated and import. Gap and tamper records only ever have source: signer; no request can carry an event type or a source.
| Group | Main event types |
|---|---|
| Lifecycle | run.registered, run.closing, run.final, run.child |
| Calls | policy.decision, tool.result, model.exchange, input.observed, state.write |
| Approvals | approval.request, approval, approval.consumed, .expired, .refused, .binding_mismatch, .escalated, .revoked |
| Keys and log | signer.epoch, key.retire, log.closed |
| Integrity | capture.gap, trace.tamper, reconcile.*, refusal.summary, policy.external |
| Effects and content | effect.observed, content.accessed, content.erased |
| Containment | incident.*, run.held, run.resumed, run.quarantined, identity.suspension |
| Analysis and other | review, why.finding, constraint.added, ext |
5.3 Commitments and the content vault
Records should be safe to hand to an auditor, but tool arguments and outputs carry secrets. A plain hash doesn't solve this: the hash of a short value, such as a PIN or the word "yes", can be guessed by trying every candidate.
- Keyed commitments. Records hold
HMAC(HMAC(args_salt, label), digest), with a salt per record.signer revealopens exactly one record for an auditor, and no other. - Signer-side redaction. The signer redacts every result itself, even when the client says it already did. A redaction manifest records the rules and counts, never the values, and flags
client_redaction_incompletewhen the client's redaction missed something. - The vault. When it is on, content is stored separately: AES-256-GCM per record, under a random per-run key wrapped by
content.key. The associated data binds each blob to its log, run,seqand commitment, so a blob can't be moved to another record. A failed save means "not retained", never that the call failed. - Reading and erasing.
content_getneeds a grant that is never default, writes a signedcontent.accessed, and returns the salt only when the content opens the commitment and isn't guessable.content_erasedeletes the wrapped key and the blobs first, then signscontent.erased. Records and chains are untouched, so old bundles still verify.
Evidence and content have opposite needs. Evidence must be permanent and shareable; content must stay private and be erasable. Commitments let a record prove what a call carried without carrying it, and let content be erased without breaking a single signature. The limits: whoever holds content.key and the store can read the vault, erasure doesn't reach backups, and timing and sizes still leak.
Code: service (_retain, content_get, content_erase, _seal) · docs/privacy.md · docs/format-v2.md
6 Making the log hard to rewrite
A hash chain shows that records weren't changed after the fact. But whoever holds the signing key can rewrite the whole chain from any point and sign it again. And a bundle for one run can't show which other runs were left out of an export. Tracekit closes both gaps with the design Certificate Transparency and Sigstore use, with one difference: the log's own operator is treated as a separate adversary.
6.1 The Merkle tree and run-sets
- Each record hash is a leaf of an RFC 6962 tree. Leaves are
SHA256(0x00 ‖ data)and nodesSHA256(0x01 ‖ left ‖ right); the prefixes stop a node being passed off as a leaf. - An inclusion proof shows a record is in a signed tree. A consistency proof shows a newer tree extends an older one, which is how witnesses refuse forks and rollbacks.
- The tree is stored as tlog tiles: its hashes cut into fixed files of 256, in the layout specified by C2SP (the Community Cryptography Specification Project, which also specifies the checkpoint and witness formats below). Full tiles never change, so a proof for an old leaf reads a few tiles. Tiles are storage, not evidence: a bundle carries proofs, not tiles. A read-only API serves checkpoints, tiles, registry logs and anchors, and
/logs/v0lists the logs for witness software to poll.
Registry logs and run-sets. Each tenant has its own registry log. Its 89-byte leaves record the type, a salted hash of the run id, the log id, the seq and the record hash. Run starts and finals get a leaf in their tenant's registry; log closes, key retirements and tenant-level gaps get a leaf in every tenant's registry. The registry's origin is derived from the tenant salt, so it never carries the tenant's name.
A run-set bundle carries the registry leaves between two registry checkpoints, with a consistency proof. The verifier checks that the leaves are exactly the ones in that range, that each points at a real record in the record tree, that each run has one run.final and all its records, and that every key retirement is present. A deleted run, a second final, or a withheld key retirement shows up as a missing or extra leaf.
6.2 Witnesses, Rekor and monitors
The operator holds the log key, so the operator could rewrite history and sign it again. Witnesses close that gap. A witness keeps the size and root it last cosigned, and cosigns only trees that extend it. A rewritten past can't get a second cosignature.
Figure 11: At startup the signer asks each witness for its latest cosigned size. If the local log is shorter, it writes a signed, tenant-level trace.tamper{rollback} and refuses client writes until an operator acknowledges it. Code: tlog_witness.py, service (rollback check, witness workers), format/checkpoint.py.
Checkpoints. A checkpoint is a C2SP signed note: origin, tree size and root, signed by the log key. Using the standard format means existing witness software (omniwitness, litewitness) can cosign Tracekit logs without knowing anything about agents. The signer writes one after each run.final, on a nudge at most once a second, every 10 seconds while the tree grows, and at close. A witness sees only sizes and roots, never records.
Cosigning. One worker per witness posts each new note with a consistency proof from the size that witness last cosigned. After 5 minutes without a cosignature it writes a witness_failed gap, one per witness for each tree that witness hasn't cosigned (the record log and each registry log). A cosignature timestamp too far from the signer's clock writes a clock_skew gap.
Quorum. Witnesses are classed public, customer, tracekit or operator, and the class comes from the verifier's trust config, never from the bundle. Each witness is pinned as one key under one name with one class, so it counts once. witnessed assurance needs at least max(1, witnesses_required) cosignatures or anchors from non-operator witnesses.
Rekor and time. Record timestamps come from the signer host's clock, and the chain proves only order. At most once an hour the signer anchors a note to Sigstore's Rekor v2 with a P-256 key (Rekor v2 doesn't accept Ed25519). The anchor's time is an RFC 3161 timestamp, which gives independent time evidence. The verifier checks all of it offline against a pinned trusted root, never the certificates inside the token.
Monitors. A monitor rebuilds the tree from the tiles, checks both chains and every run's start and final, and signs a report. A fresh, conflict-free report from a pinned monitor lifts witnessed to witnessed+monitored. That catches an operator showing two different forks to two different verifiers at the same tree size.
Limits. A log never shown to a witness isn't protected by one. A split view needs verifiers that pin different witnesses or accept less than witnessed, and a monitor catches it only at equal tree sizes. Our public witness server isn't deployed yet. Rekor entries are public and permanent, and reveal timing.
7 Completeness: signing what is missing
Signatures prove the records weren't changed. They can't prove the records are everything that happened. A missing record looks exactly like nothing happening. So Tracekit treats completeness as something to measure and report, never to claim. The signer compares independent views of the same run and signs every mismatch.
Reconcile compares three layers. L1 is saved agent state (state.write). L2 is what the signer decided and recorded. L3 is the tool uses in what the model said (model.exchange), taken from the model client or, better, a mandatory LLM gateway. Items pair by tool call id. When an adapter has no ids, a decide and a tool use pair only inside the same model turn, with equal argument digests, and only when the match is unambiguous; never across turns.
Figure 12: Reconcile. Other outcomes: result_without_call, args_unparseable, result_mismatch, and unexplained_effect, an effect an observer saw with no call behind it. With gateway_mandatory, only the gateway's L3 counts. Code: signer/reconcile.py (_found, finish).
| Outcome | Meaning |
|---|---|
hook_missing | The model asked for a call that was never decided: the agent acted outside the gate |
fabricated | A decided call that no model message asked for |
args_mismatch | The tool or arguments differ between what was asked and what was decided |
args_unparseable | The model's raw arguments failed strict parsing |
result_without_call | A result reached the model for a call nothing recorded |
result_mismatch | The result the gateway saw go to the model isn't the one recorded |
unexplained_effect | The egress observer saw a host that no recorded call, nor the policy, accounts for |
Most outcomes are written at run close, just before run.final. Three are written as they happen: result_mismatch and unexplained_effect always, and hook_missing when a gateway sends a call's result back to the model with no decide behind it. An imported decision that disagrees with the signer's is not a reconcile outcome: it is a capture.gap{decision_mismatch}, written at run close.
Every gap is a record: chained, signed and counted, so deleting one breaks the run chain. Only the signer writes gaps; no request field can carry one. Tenant-level and late gaps also get registry leaves, so a run-set proves them. The verifier's gaps line counts them.
| Written by | Gap kinds |
|---|---|
| The pipeline | client_counter_gap, concurrent_run, signer_unavailable (written first on recovery) |
| Decide and complete | executed_against_policy, class_mismatch, decision_flip |
Approval requests (approval_request) | reask_after_reject |
Model exchanges (model_event) | result_unchecked |
State writes (state_write) | state_tamper |
Constraints (constraint_add) | constraint_refused |
| Run close | decided_not_completed, held_call_unresolved, late_activity, log_closed, decision_mismatch |
| Observers | tailer_lost, tailer_redelegated, observer_lost, effect_outside_run |
| Tenant level | witness_failed, clock_skew, degraded_unanchored, trace.tamper (rollback) |
| Store checks (signer-level, not tenant-level) | trace.tamper (edited) |
Two limits matter here. Kernel sensors (Tetragon- and Falco-shaped events) are recorded as the sensor's word, not reconciled by the signer; tracekit analyze --v2 checks them instead. And an agent that never sends its model's messages leaves L3 empty. In that case nothing is compared, and the report says so instead of showing a clean result.
8 The bundle and verification
8.1 The .tkb bundle
Evidence has to travel to someone who doesn't trust the sender, and be checkable with no network and no access to the signer. A .tkb is a zip with one canonical layout: manifest.json, runs/, keys/records.jsonl, proofs/records.json, checkpoints/<size>.note, optional rekor/ and tsa/, and policies/<sha256>.json. A run-set adds the registry files. File names use hashed run ids, so they never carry caller text. Anything that isn't the canonical layout is UNUSABLE BUNDLE, so every reader sees the same entries. Caps: 10,000 entries, 64 MiB per entry, 512 MiB in total. A run over 64 MiB is split into parts and checked as their concatenation; a run cut short reads as open, never plain VERIFIED.
Four things are absent on purpose:
- No code. The party being checked must not supply the checker.
- No trust config. The person verifying brings their own.
- No vault content. Records carry commitments, so a bundle can be shared.
- No trusted manifest. The manifest is an index, never a source of trust.
Format versions. Each format version lists the record types, fields and layouts it added. From that, a bundle states the oldest verifier that can read it. An older verifier then says UNVERIFIABLE (needs tracekit ≥ x), never FAILED, for a record type it has never seen. A version's row is frozen once its first release candidate ships, and old verifiers are tested against new bundles. Evidence must stay checkable for years.
8.2 What a verifier checks
A verifier trusts only the trust config it is given: log keys and algorithms (required), and optionally witnesses with their classes and quorum, Rekor, monitors, issuers and analyzers. No other key is accepted. Anyone can sign a bundle and a checkpoint with keys of their own, so the trust file is what says whose signatures you accept. A trust file handed over by the operator proves nothing about the operator; an auditor should obtain each key independently.
- Load the trust config. If it can't be used, the answer is
UNVERIFIABLE. - Read the zip, check the manifest format and the version gate, and check that the manifest's hashes equal the files present, exactly.
- Open the checkpoint with the pinned log key. Its size must equal the proofs' size.
- Count witness cosignatures against the quorum, then check Rekor anchors and monitor reports.
- Build the key timeline from
signer.epochandkey.retirerecords, each included in the checkpointed tree, with certificates checked against pinned issuers. - For every record of every run: the signature under the key valid at its
seq, the schema,log_id, the run chain, and inclusion of the first and last records. - Check the bundle's shape: one run, one run with its findings, or a run-set with its registry checks.
- Check the policy snapshots against the hashes the decisions name.
- Give the integrity verdict, then the warning and information lines: policy, held calls, gaps, coverage, matrix, observers, findings, reviews, constraints, incidents, tiers, isolation, harness, fail-open classes, key assurance, approvals and break-glass.
| Verdict | Meaning | Exit |
|---|---|---|
VERIFIED | Every check passed. An open run reads VERIFIED TO HEAD n; a run-set with open runs reads VERIFIED (k open) | 0 |
FAILED | The evidence is bad: a signature, chain, inclusion or quorum check failed | 1 |
UNVERIFIABLE, UNUSABLE BUNDLE | This verifier can't judge: a bad trust file, a version it doesn't know, a revoked key, or a feature it doesn't implement | 2 |
--strict | Any warning fails the run | 3 |
Assurance is reported separately from integrity, as dev, local, witnessed or witnessed+monitored. Any self-approval in the run caps it at dev. With the dev signer, findings read as self-asserted and isolation as same-user. No verifier says VERIFIED for a check it skipped: where an implementation lacks a feature, it answers UNVERIFIABLE instead of passing. There are two exceptions. Records carrying an ext name a verifier doesn't interpret verify like any other, with a not interpreted warning, by design. And a verifier older than 1.4 doesn't read signed findings (see Section 12.1).
8.3 Three verifiers
A bug in one verifier must not decide an audit on its own. There are three implementations. Python is the reference. TypeScript was written from the format spec rather than from the Python code, and a test requires it to agree with Python on integrity, assurance, exit code and every check line. Go is experimental, and was written while consulting the Python verifier and the golden corpus, so it doesn't show that the format spec alone is enough to write a verifier. Its own gate compares exit code, integrity and assurance with Python's. The golden corpus is frozen and regenerated only by hand; it includes 48 bundles with one defect each and 7 defective multi-part runs.
A browser page runs the TypeScript verifier in a worker. Its content security policy is default-src 'none', with scripts and styles allowed by hash and connect-src 'none', so it can't load or send anything. You can check your copy against the release with gh attestation verify.
Code: verify/v2.py (verify, load_trust) · sdk/typescript/src/verify/ · contrib/independent-go-verifier · tests/golden/ · docs/verdicts.md
9 Reading the record
A verified run is a set of facts. Several tools read those facts and add opinions on top. The rule throughout is that an opinion never changes integrity. tracekit analyze, tracekit why and tracekit view read a bundle's runs only once it verifies (analyze can also read the signer's own store). tracekit graph is the exception: it keeps a bundle that fails, and grades every edge from it UNVERIFIED.
Signed findings. tracekit analyze --v2 reads the signed records and, from the vault, what they commit to; each item must open its commitment or the analysis stops. Findings are written into a separate findings run, and only identities listed as analyzers may register one. The signer refuses a finding outside a findings run, or one that cites anything but a record of the analysed run with that exact hash, and writes the finding's title itself. The verifier labels each finding as independent, unpinned author, self-asserted or author not checked. A missing finding proves nothing.
| Id | Finding | Severity |
|---|---|---|
TK-X201 | Action with no reasoning | info |
TK-X202 | Edit of a file never mentioned | medium |
TK-X203 | Write outside the project | high |
TK-X204 | A stated constraint broken | high |
TK-X205 | Success claimed after a failure | critical |
TK-X206 | Effect with no action | high |
TK-X207 | Action with no effect | medium |
TK-X208 | Prompt the harness never submitted | high |
TK-X210 | Flagged by the reviewer model | warn |
tracekit why asks which input caused an action. It doesn't trust the model's explanation of itself. It replays and measures, and what it may claim depends on how far up this ladder it went:
| Rung | Method | What it can claim |
|---|---|---|
| 0 | Observed provenance, or a predicted ranking | Never "tested" |
| 1 | Replay one recorded model call with one item removed | Tested, for that call |
| 2 | Replay the run without the agent, tools answered from the run's own tape | Tested, for the run |
| 3 | Replay the program | Tested, for the program |
Each finished test is signed as a why.finding, and the verifier recomputes its counts, interval and verdict from the bundled replay runs.
tracekit graph (experimental) builds a graph across bundles, verifying each against the trust config, and grades every edge SIGNED, OBSERVED, INFERRED or UNVERIFIED. A bundle that fails verification stays in the graph with unverified edges, and a why test is drawn as causal only when the verifier counts it. The graph file itself is unsigned; tracekit graph verify rebuilds it to check.
tracekit view runs tracekit verify's own code on each run before showing it. A failed run shows only its report, and a run no checkpoint covers yet reads PENDING. Its screens are Desk (Needs you, Span), Runs, Live, Approvals, Permissions, Horizon (the project, this machine, the outside world) and Influence.
Proof-carrying messages (PCM). When one agent tells another "tests passed", the receiver has no reason to believe it. The signer signs a PCM message only as the calling run's agent. The message carries a typed intent, claims with evidence, and grants. Four claims can be proven: tests_passed, approved_by, reached_host and file_written. Each counts as proven only through records the signer or a T2 path wrote; a T1 result stays unproven. The evidence is a pinned, cosigned checkpoint, record hashes, inclusion proofs and arguments opened by their salt. A claim proves a call happened, not that nothing happened after it, and grants can't be revoked yet.
10 Experiments
These are our own experiments: small, synthetic and written by us. No one outside has reviewed the threat models yet. We report the misses alongside the hits, because the misses show where the design's edges are.
10.1 Tamper detection (E1)
We built 120-record signed, hash-chained logs in the v1 JSONL ledger format and ran eight attacks against them, 60 trials each. The signed log alone caught six of the eight in every trial. It caught truncation of the tail and re-signing with the key in none. That is the expected result: a log can't see its own tail cut off, and whoever holds the key can re-sign it. With a witness checkpointing every 10 records, all eight were caught in all 60 trials.
Figure 13: Trials caught out of 60, per attack. The two misses of the log alone are the reason witnesses exist: an unwitnessed run therefore verifies at assurance dev. Source: E1 (eval/e1_integrity.py), which tests the v1 JSONL ledger, not the v2 Merkle log. Its results were committed on 2 October 2026, before 1.2. The script is seeded, and CI re-runs it and diffs its output against the committed results, so the numbers still reproduce.
Checkpoint spacing is the attacker's window. Checkpointing every 1, 5 or 10 records caught every rewrite; every 25 records caught 90%; every 50 or 100 caught 82%.
Figure 14: Rewrites caught as checkpoints get further apart. The signer's default writes a checkpoint at most once a second, every 10 seconds while the tree grows, and after each run.final.
10.2 All experiments
| Experiment | Setup | Result |
|---|---|---|
| E1 tamper detection | 120-record signed logs in the v1 JSONL ledger format, 8 attacks, 60 trials each | 6 of 8 caught by the log alone; 8 of 8 with a witness |
| E8 insider attacks | Real unprivileged users against a Linux system-mode signer: a decoy signer, a stopped signer, a swapped policy, fabricated runs | 8 of 8 caught on the v1 signer, 6 of 6 on v2 (including self-approval); a merge gate on every pull request |
| E9 overhead | Signer round trip over a Unix socket, file storage, macOS | 0.45 ms p50, 0.91 ms p99 per call; 5,356 events/s (a quick macOS run committed on 10 October 2026, before 1.0) |
| E10 reconciliation | 30 runs, 12 with a seeded discrepancy: a missing hook, a fabricated call, changed arguments | 12 of 12 flagged, 0 false positives (with L3 present) |
| E11 saved-state tampering | Edits to the agent's saved state | 21 of 21 caught |
| E12 tenant isolation | Cross-tenant attacks under concurrency | Every attack refused, no leaks |
| E14 disk full | Storage fails mid-run | 41 calls acknowledged, none lost; two gaps written: signer_unavailable, and a client_counter_gap covering the 22 refused calls |
| E17 policy packs | 766 harmful and 834 benign tool calls in 11 classes, decided by both regex engines | 766 of 766 denied or held; 1 false alarm (a draft email held for a person) |
| E18 agent approvers | Attacks on the agent-approver path | 7 of 7 caught |
| E21 policy robustness | 420 harmful and 432 benign calls written blind to the rules | Server pack: 211 of 420 stopped or held on the one blind run; 18 of 432 false alarms; p99 125 µs on the blind run (19 October 2026; 185 µs in the results committed on 30 October 2026) |
| E3 the v1 regex gate | 16 harmful and 20 benign calls written after the rules | 11 of 16 caught, 1 false alarm. Rules are tripwires, not a sandbox |
E21 is the number we watch most closely. On calls written without seeing the rules, the server pack stops or holds about half the harmful ones. A regex gate is weak as prevention: a determined agent can reword a command. It is deterministic, though, so a verifier can recompute every decision, which a model-based judge doesn't allow. Tracekit's contribution is evidence that the gate ran and what it saw. For preventing effects, pair it with a sandbox.
11 Design decisions and invariants
| Decision | Reason |
|---|---|
| A separate signer process | Who can rewrite the log decides what a run is worth. Only the signer holds keys, sequence numbers and chain heads |
| Certificate Transparency's design and C2SP formats | Existing witness software, and a format precise enough for an independent verifier |
| Keyed commitments, not plain hashes | Short values can't be guessed, and one record can be opened on its own |
| Deterministic rules, not a model judge | The same policy, engine and arguments give the same decision, so it can be recomputed. The one exception: a regex timeout on the fallback engine denies the call and marks the decision nondeterministic |
| Fail closed by default | An early insider test showed an agent could stop the signer and act unrecorded (fixed for system mode in 0.2.1) |
| No verifier inside a bundle | The party being checked must not supply the checker or the trust |
| Frozen verifiers and a version gate | Evidence must stay checkable for years; an old verifier says "needs newer", never "failed" |
| Run tokens, not keys | A capability for one run, in a key domain apart from evidence signatures |
| A chain per run and one for the log | The log chain orders everything; the run chain makes one run's bundle provable with two inclusion proofs |
The code keeps eleven invariants, I0 to I10, each mapped to tests in tests/INVARIANTS.md:
- I0. The agent can't reach the signer's store, keys, witnesses, code, config or policy, except in dev mode, which says so.
- I1. The agent never holds a signing key or a chain head, and never assigns sequence numbers. A client's own counter is used only to detect gaps.
- I2. Policy runs on the raw arguments, with the signer's own tool classes, deterministically.
- I3. Approvals bind to exact arguments and are used once.
- I4. Runs are registered before their first action and end with a signed
run.final. - I5. Identity proves the workload, not the tenant or user: a tenant and a principal are attested, or labelled as asserted by the app.
- I6. Gaps are signed, never silent. Clients can't forge gap or tamper records.
- I7. Signatures name their algorithm and key and are domain-separated; key retirement is positional.
- I8. Old evidence stays verifiable: verifiers are frozen per format version, and the golden bundles they are tested on stay frozen.
- I9. The verifier and its trust config never come from the party being verified.
- I10. Canonical JSON is strict: no number overflow, non-finite numbers, duplicate keys or lone surrogates.
The same rules read from the other side, as what the signer refuses to take from a caller: its identity (the transport sets it, on Linux for every frame); an event type, source, isolation level, fail mode, or gap or tamper record; the tool's class (the policy maps it, and a hint can only tighten); the approver's identity; a run token for another run; and a gateway answering for another caller's call. Posture and principal claims are recorded, but marked as not attested.
12 Known limits and open problems
Tracekit proves what its capture path recorded, and that the record has not changed since it was signed and checkpointed. It doesn't prove intent, completeness, or that a reported result is true, and it detects tampering rather than preventing it. Within that claim, these are the seams we know about.
12.1 Seams in the code today
- A decision's
expires_at(300 s) is signed but not enforced bycomplete. The code marks this with alean:comment. - The set of denied argument digests behind
decision_fliplives in memory, so a flip across a signer restart goes unflagged. - A renamed tool, such as CrewAI's
bash, is decided under its listed spelling too. Constraints, egress, detection sums and the outage fail mode still go by the name as sent. - A bundle's manifest is unsigned. Editing its
verifier_min_versionlets a verifier older than 1.4 check a bundle it can't fully read: it still checks every signed record but skips signed findings. Verify with 1.4 or newer. - Containment is enforced only on T2 paths (Section 4.7).
content.keyhas no rotation path and isn't KMS-wrapped, and vault erasure doesn't reach backups.- The TypeScript verifier reports SLH-DSA lines, Rekor anchors, record-key certificates and
tracekit whyfindings asUNVERIFIABLErather than checking them.
12.2 Who watches the signer?
Witnesses the operator doesn't control, a monitor run elsewhere, and Rekor anchors. They detect a dishonest signer after the fact. Preventing one would need attested hardware, which Tracekit doesn't claim. Without independent witnesses, an operator rewriting its own log is expected behaviour, not a vulnerability. Our public witness server isn't deployed yet, and when it is, it won't be independent of Cygnux Labs. Independent witnessing that needs no setup is the largest open item.
12.3 What if the agent never calls the hook?
Then nothing records the call. With its run token, an agent can also report well-formed calls that never happened, and the signer can't tell them from real ones. The answer in both cases is a layer outside the agent's process: a mandatory LLM gateway, the executor or MCP gateway holding credentials, or the egress observer. Capture of services, and of processes beyond the experimental kernel sensor import, is not done yet.
12.4 What leaks?
Arguments and results are commitments, but timing, sizes, counts and checkpoint cadence still show. Rekor entries are public and permanent. Redaction is pattern-based. A run-set bundle carries the tenant salt, so whoever holds it can test guessed run ids for that tenant.
12.5 Evaluation
The threat models haven't had an external review. The experiments are small, synthetic and written by the project, and the attacks are scripted rather than run by real models acting adversarially. System mode needs validation on macOS and Windows, and we don't yet have performance numbers for separate-user and HTTPS signers on Linux. Each of these is work we want outside help with.
13 Conclusion
Most of Tracekit is not new theory. It applies ideas that already work elsewhere to a new kind of actor: Certificate Transparency's witnessed Merkle logs, the single-writer discipline of databases, keyed commitments from cryptography, and a strict separation between the party being checked and the code that checks it. What is specific to agents is the decision path in front of the log, the custody paths that move credentials out of the agent's process, and reconciliation that turns missing activity into signed records.
The result is a record whose value is stated rather than assumed. A verifier tells you how isolated the signer was, who vouched for the checkpoint, which calls came through paths the agent controlled, and what is missing. As agents take on more work with less supervision, we think every team running one will need a record like that, and we would like it to be an open standard rather than a product feature.
Try the full stack in one command, with no API key:
uvx --from tracekit-ai tracekit demo --server
# or add it to your own agent
pip install tracekit-ai
Tracekit is open source under the MIT licence: github.com/Cygnux-Labs/Tracekit. The concept pages in the wiki cover each part above in more depth, and the limits page lists every known gap with what to do about it. We welcome reviews of the threat models, attacks on the design, and pilots on real agents.
This article describes Tracekit as it is on main on 2 November 2026, after release 1.5.2. Facts come from the source code and the project's design documents: docs/concepts/, docs/format-v2.md, docs/verdicts.md, docs/limits.md and the laptop and server threat models. Rule counts are unique id: values across tracekit/policy2/packs/*.yaml. Experiment results are from the project's own evaluation and README.
Cygnux Labs · github.com/Cygnux-Labs/Tracekit














