09How does an engine batch requests for different LoRA adapters in one forward pass, and what do Punica-style shrink and expand kernels do?▼hardNewFireworks AITogether AIDatabricks○ sign inEight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.Open full answer →
07Design serving for 500 customer LoRA fine-tunes of one base model. Where do the adapters live, and how does a request find one?▼hardNewFireworks AITogether AIModal○ sign inFive hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.Open full answer →