cost per token

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

Offline work changes the regime: the batch is as big as memory allows, latency is irrelevant, and output tokens stop being expensive. The answer prices both passes in GPU-hours, finds which one is the bill, and cuts it.

A cheaper GPU and a faster decoder do not automatically produce a cheaper service. Audit a cost ledger, reject misleading wins, and choose a rollout only after quality and latency gates pass.