Skip to content
← Inference System Design / 07
hardNewFireworks AITogether AIModal

Design serving for 500 customer LoRA fine-tunes of one base model. Where do the adapters live, and how does a request find one?

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackDesign a multi-region deployment for a 70B model that survives the loss of a whole region. How much spare capacity do you buy, and where?Next in this trackDesign a platform serving 20 models of different sizes on one shared fleet. How do you place them, and when does a model get its own GPUs?Next in this trackPrice an LLM API: go from a GPU-hour cost to a per-million-token price for input and for output. Why do they differ?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.