What it does
Several services in Vision talk to language models: the chat agent, the eval harness, the indexer. None of them holds a provider key or names a model. They all send their requests to one LiteLLM gateway that I host myself, and ask for a role instead of a model: smart, cheap, local or embed.
The gateway decides which concrete model sits behind each role, knows what every call costs, and stops a flow that has spent its share. That gives me one place to switch providers, one place to read the bill, and one place where a runaway loop hits a wall.
Why this way
Decision 01
One gateway, roles instead of model names
A service asks the gateway for smart, cheap, local or embed, never for a provider’s model. One config file maps each role to exactly one model. Changing provider means editing that file; no service changes, and no service needs a key of its own.
Rejected: a provider SDK and key in every service
Each service would carry a key that can spend money, nothing would add their spending up, and every provider switch would touch code in several places.
Decision 02
Prices I set myself
The config lists candidate models per provider and tier under stable names, and every role points at one of them. I write each price into the config by hand, with the date I last checked it, and LiteLLM’s downloaded price list is turned off. Prices are checked again whenever the gateway is upgraded or a candidate changes.
The rule I’m working towards: each role maps to the cheapest candidate that passes the evals for that kind of task. Today’s mapping still reflects which provider key I set up first, and it is written down as provisional.
Rejected: trusting LiteLLM’s own price list
The pinned version had no entry for the newest models, and later releases contradicted each other.
Decision 03
A key and a budget per flow, under one global cap
The chat agent, the eval harness and the indexer each get their own virtual key, a 30-day budget, and only the roles they need. A global cap sits above all of them.
The cap has to survive a calendar mismatch: LiteLLM resets budgets on the 1st, while my provider’s billing cycle starts on another day, so one billing cycle can see two full LiteLLM budgets. Against my monthly API credits I first set the cap at $45, so even the doubled worst case of $90 stayed inside them. Since 2026-11-05 a $30 monthly spend limit at the provider is the hard backstop, and since 2026-11-06 the global cap is $24 and the key budgets add up to $22.
Rejected: a cap of about $90 to match the credits
It only holds if LiteLLM’s month and the billing cycle start on the same day. They don’t.
Decision 04
Running out of money is its own outcome
Used-up provider credits and an exceeded LiteLLM budget look the same to someone chatting, but they have different fixes. agent-core tells them apart: the run fails with a named reason, the reply says what ran out and what still works, and the status command keeps showing it until a call succeeds again. Nothing retries and nothing switches provider. local keeps answering, because it isn’t paid from the credits.
Rejected: a generic “sorry, that failed”
It hides a problem I could fix in a minute.
Decision 05
One exception: my subscription, for my own messages
A Claude Code worker runs on my Claude subscription instead of the API. After reading the terms I limited it to requests I send myself with an explicit command. It never runs unattended, makes at most 30 calls a day and starts every run in a clean workspace. It is kept off the gateway’s network: a filtering proxy is its only way out, to the provider’s API and the gateway’s tool endpoint. It is the only model traffic that doesn’t pass the gateway.
Rejected: the subscription as the default for complex messages
That was my first design. Once I had prepaid API credits, complex messages went to smart through the gateway instead, where budgets, chat history and evals apply.
Decision 06
Prompt caching on the stable part of every request
For smart and cheap, each request marks its stable start, the instructions and tool definitions, and its growing end. Every call in a tool loop then reads the previous call’s prompt from the provider’s cache instead of paying for it again. Cache reads and writes are recorded per step, and every cached step’s cost matched its tokens times the cache prices I pinned in the config.
How it's tested
Each way of running out gets its own reply
Tests cover both cases, used-up credits and an exceeded budget, and check that each produces its own outcome and its own reply instead of a generic failure.
A fix counts only after a cold reboot
For the start-up problem below, a check script tests name resolution in every container. The fix was accepted only after a cold reboot with no manual restarts, because a warm restart had hidden the problem before.
What it costs
The first 31 real chat runs cost $0.05 together. A complex question with six tool calls on smart cost $0.107, and a homelab status question $0.018. On top of its key’s budget, the eval harness stops any single run at $2.
Prompt caching cut the agent eval’s cost per case by 65 %, from $0.040 to $0.014, with the same quality.
There is no with-and-without number for the gateway itself: what it buys is control, not speed.
What I learned
Case 01
A budget is only as good as its price table
What happened
When I first wired things up, the pinned gateway didn’t know the prices of the models I used. Every call would have been logged at $0, and no budget would ever have stopped anything.
What I changed
All prices now live in the config, copied by hand from the providers’ pages, and LiteLLM’s own price list is turned off.
Result
The budgets rest on prices I set myself, and those are re-checked on every upgrade.
Case 02
Every chat failed after a reboot
What happened
One morning no message got through: the gateway couldn’t look up the provider’s address. Docker hands each container its name servers when the container starts, and at boot some containers started before the host had its own settings, so they got none.
What I changed
A restart fixed it until the next boot, and reordering the services would only have made the race rarer. Docker now uses fixed upstream name servers.
Result
A check script tests name resolution in every container, and the fix counted only after a cold reboot with no manual restarts.
Case 03
A fallback I couldn’t see
What happened
The gateway can switch from one role to another on its own. For the classifier that meant local could time out in agent core while the gateway quietly retried on cheap: money was spent that the run never recorded, and the run no longer matched the spend log.
What I changed
I turned the gateway’s fallback off for that call, so only agent core decides when to fall back.
Result
Every call now has to appear in both the run record and the gateway’s log, joined by run and step number.