03Why do serving engines capture a CUDA graph per batch size, and what does padding to the nearest captured size actually cost?▼mediumNewNVIDIAFireworks AITogether AIunlockedA graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.Open full answer →
07TensorRT-LLM in 2026: what is the PyTorch backend, how does its in-flight batching admit requests, and when do you still pick it?▼mediumNewNVIDIAPerplexityBaseten○ sign inMost answers still describe building a TensorRT engine per GPU and batch shape. The current release accepts only the PyTorch backend, and its default admission policy can halve your concurrency. The answer covers both, then says when it is still the right engine.Open full answer →