BROWSE BY CONCEPT

Questions by tag

kv cache49 decode34 vllm26 debugging21 speculative decoding20 capacity planning20 coding19 ttft17 roofline15 prefix caching15 fp815 attention12 long context12 prefill11 quantization11 numerics11 memory bandwidth10 tensor parallelism9 expert parallelism9 preemption9 gqa8 moe8 cost per token8 tpot8 scheduling8 flashattention8 int48 benchmarking7 observability7 paged attention7 disaggregation7 continuous batching7 chunked prefill7 streaming7 gpu memory6 throughput6 batching6 cuda graphs6 capacity6 hopper6 acceptance rate6 eagle6 mla5 profiling5 latency5 copy on write5 scheduler5 admission control5 async scheduling5 triton5 routing5 vllm metrics5 mbu4 kernel fusion4 inter token latency4 token budget4 slo4 cpu overhead4 tiling4 split kv4 gemm4 blackwell4 calibration4 nvfp44 sampling4 sglang4 structured output4 all reduce4 nvlink4 eviction3 radix tree3 littles law3 tail latency3 utilization3 rate limiting3 batch inference3 online softmax3 causal mask3 l2 cache3 tma3 warp specialization3 wgmma3 cuda3 shared memory3 evaluation3 determinism3 kernels3 rejection sampling3 constrained decoding3 xgrammar3 pipeline parallelism3 all to all3 deepseek3 rdma3 collectives3 load balancing3 testing3 arithmetic intensity2 gpu memory utilization2 sliding window2 hybrid attention2 kv offload2 pcie2 interconnect2 system design2 memory budget2 goodput2 multi tenant2 fairness2 model runner2 io complexity2 logsumexp2 flash decoding2 tensor cores2 rope2 bf162 embeddings2 context parallelism2 gemv2 occupancy2 h1002 performance debugging2 awq2 weight only quantization2 gptq2 fp42 block scaling2 kl divergence2 mixed precision2 deepseek v32 llama cpp2 batch invariance2 temperature2 top p2 greedy decoding2 medusa2 draft model2 grammar2 beam search2 micro batching2 multi node2 kv transfer2 cold start2 autoscaling2 startup2 tensorrt llm2 detokenization2 lora2 multi lora2 multi tenancy2 tensor parallel2 nccl2 block manager2 cancellation2 request lifecycle2 correctness2