sampling
Inference interview questions tagged sampling, across every topic.
4 questions · 3 unlocked for you
Concepts behind "sampling"
Concept pages for the mechanisms these questions lean on.
Advanced
Grammar-Constrained DecodingConstrained decoding masks every token the grammar cannot accept before sampling, using an automaton compiled from a regex, JSON schema or context-free grammar. This page covers how the mask is computed fast enough to hide under a 5 ms decode step, why per-step masking does not sample from the model conditioned on the grammar, and how it interacts with speculation.🌳 Decoding Algorithms🔒 Premium
Advanced
Sampling: Temperature, Top-k and Top-pHow LLM token sampling works and how to implement it correctly: temperature rescales logits, top-k keeps the k largest, top-p keeps the smallest prefix of the sorted distribution reaching probability p. Covers order of operations, ties at the cutoff, temperature 0, fully masked vocabularies and the cost at full vocabulary size.⌨️ Implementation Patterns🔒 Premium