What it does
Every service in Vision that does work writes the same kind of record: the chat agent, the indexer, the MCP servers, the eval harness and the cleanup jobs. A run stands for one piece of work, and every model call, tool call or command inside it is a step with its tokens, cost and time. One query then answers what ran yesterday, what it cost and what failed, across everything.
The records hold names, counts and costs. The words themselves, the question, what each step saw and the answer, live in a separate trace store for the console, and that store forgets them after 30 days.
Why this way
Decision 01
One shared library, one contract
A small shared library opens a run, adds its steps and closes it with totals, and every service that does work uses it. A tool call made through the gateway becomes a child run of the run that made it, so a tool loop stays one tree.
Decision 02
One step per model request, tagged so it joins the gateway’s log
Each request to the gateway is one step, tagged with its run and step number. My records and the gateway’s spend log then join exactly, even inside a tool loop where one message makes several model calls. A step records its tokens, its prompt-cache tokens and its cost. The subscription worker’s steps keep the cost its tool reports in a column of their own, so it never counts as API spend.
Rejected: the gateway’s spend log on its own
It can’t see commands or tool calls, or which message caused a call.
Decision 03
Refused is not failed
A run stopped by the kill switch or by a session’s budget closes as cancelled, with the reason, not as failed, and a flow that is switched off writes no run at all. The weekly failure counts then show real failures only. A request from an unverified sender is the exception: that one is recorded as failed, with the reason.
Decision 04
Counts in the record, content in a store that expires
The run records hold names, counts, costs and timings. The full content, meaning the question, each step’s input and output and the answer, lives in a separate trace store for the console, and is deleted 30 days after the run unless I flag it; a flagged run stays until I mark it handled. The question in the run itself is cleared after the same 30 days, and only a hash of it stays for checking where requests came from. Error text in a record is cut short, and the full error expires with the trace. Tool calls and eval runs keep the tool’s name and settings, never the query, because in an eval the query is a held-out question.
Rejected: keeping chat history with no expiry
That was the original plan. Everything kept forever is everything at risk forever.
Decision 05
The database enforces it, not the code
Each service writes with only the rights it needs. Deleting goes through one narrow, fixed path, and the chat agent can use that path without gaining any other right to delete or change content. The readers behind the console’s overviews and the evals can’t see the question text at all.
How it's tested
Matched against the gateway’s own log
Model steps match the gateway’s spend log: tokens exactly, cost within the rounding of the stored precision. After answers started streaming, all 158 paid steps matched to within a thousandth of a cent; one eval step was off by exactly that much, because the stream’s cost sat just above a rounding midpoint. With prompt caching, the records show cache reads and writes per step, and the share of cache reads became an eval metric, so anything that silently breaks the cache shows up as a falling number.
Expiry, tested with a fake clock
A 31-day-old run is purged, a flagged one is kept, a handled one is purged, and the records of other flows are left alone.
Tests against the real permissions
Database tests run on a test database with the same rights each service has in production. For example, a run’s question can’t be changed once it is written.
What I learned
Case 01
A cancelled step lost its row
What happened
A classifier call was cancelled after its deadline, and the cancellation also hit the cleanup’s own database write, so the step’s trace record was cut off halfway. I found it while diagnosing an unrelated failure.
What I changed
Trace writes are now shielded from cancellation, and a failed trace write never fails the run; it is counted in the status command instead.
Result
A cancelled step keeps its record, and a problem with the trace store shows up as a number rather than as a broken run.
Case 02
My day ended at 1 a.m.
What happened
The worker’s daily cap and “today” in the cost command counted days in UTC, on a database that runs in UTC. My day therefore ended at 1 a.m. local time, and at 2 a.m. in summer.
What I changed
Both now count days in my own time zone.
Result
A daily cap and a daily total mean the day I actually live in.