17Implement cancellation for an inference request without reusing KV pages while GPU work still references them.▼hardNewOpenAIAnthropicNVIDIA◆ premiumA disconnect arrives after launch, then the request ID is reused. Implement deferred reclamation and prove that late or duplicate completions cannot emit stale output or free another request's pages.Open full answer →
21A streaming client stops reading. Keep its output bounded without blocking the inference batch.▼hardNewOpenAIAnthropicBaseten◆ premiumOne slow connection can stall unrelated generations or retain an unbounded output queue. Implement a byte budget that includes the pending write, then connect overflow and disconnects to engine cancellation.Open full answer →