Skip to content
AI & ML ENGINEERING

d-Matrix Inference Engineer interview questions

d-Matrix builds inference accelerators, and its LLM inference engineers prototype serving on heterogeneous hardware, tune kernels, choose quantization, sparsity and batching strategies for its compute model, and work on tensor and pipeline parallelism, disaggregated prefill and decode and KV-cache management. SIMD kernel roles write operators such as softmax and normalization.

16 questions to practise21 concepts to master7 core topicsrole: Inference Engineer

The d-Matrix Inference Engineer interview process

Limited public data
RoleSenior Staff LLM Inference Engineer
No interview process on postings.
WHAT THEY'RE EVALUATING
  • Custom kernels and operator-level optimization
  • Quantization, sparsity and batching strategies
  • Tensor/pipeline parallelism, disaggregated prefill/decode, KV-cache management
  • Mapping AI framework computational graphs onto hardware

Sources and review scope

Assembled from company hiring pages and candidate write-ups, each one listed under Sources above. A serving team, a kernel team and a research-engineering team at the same company can run different rounds, and the format shifts by level and office. Use it to plan your preparation, then ask your recruiter which rounds your loop has.

d-Matrix Inference Engineer salary

Every figure below names its source. Where no posting or aggregator gives a range for this company, the section says so and shows no number.

POSTED BY D-MATRIX
$195K - $285KbaseEmployer posting

This band covers the title Senior Staff LLM Inference Engineer. Read it against that title only. A single req often covers two or three levels, which means the upper end of a wide range is the senior hire, not the median offer.

Employer-posted base range, seen 16 October 2026; equity is additional and a bonus target is additional.

Every posted band we have, by employer and level, is in the inference engineer salary guide.

Representative Inference Engineer questions for d-Matrix's loop

None of our questions is tagged to d-Matrix yet, so these come from the tracks its loop covers, sorted by reader votes.

16 questions · 5 unlocked for you
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →
Open full answer →

Go deeper on the topics d-Matrix's loop tests

The tracks that map to a d-Matrix Inference Engineer loop, ordered easy to hard.

Quantization & Numerics15 →KV Cache & GPU Memory16 →Batching, Scheduling & Admission15 →Distributed & Disaggregated Inference15 →Attention Kernels16 →Serving Engine Internals16 →Implement-from-Scratch Coding21 →

The concepts d-Matrix's Inference Engineer loop assumes you know

What a d-Matrix interviewer expects you to reason from without being told: memory traffic, KV sizing, scheduling, parallelism. Foundation concepts are free to read, the advanced ones need Premium.

QUANTIZATION & NUMERICS

Foundational
Number Formats for Inference: BF16, FP8, INT4, FP4Inference formats differ in three ways that matter: bytes per element, which sets decode speed and memory; exponent bits versus mantissa bits, which sets range and precision; and whether the GPU has tensor cores that multiply them natively. Llama 3.3 70B is 141 GB in BF16, 71 GB in FP8 and about 40 GB in NVFP4.
Advanced🔒 Premium
Weight-Only Quantization Versus W8A8Weight-only quantization (W4A16, W8A16) stores small weights but multiplies in BF16, so it speeds up bandwidth-bound decode and does nothing for prefill. W8A8 quantizes activations too and runs on FP8 or INT8 tensor cores. The crossover batch is set by the GPU, about 141 on a B200, and activation outliers decide how hard W8A8 is.
Advanced🔒 Premium
Block-Scaled FP4: MXFP4 and NVFP4MXFP4 and NVFP4 both store E2M1 4-bit values with a shared scale per block. MXFP4 uses one power-of-two E8M0 scale per 32 elements (4.25 bits); NVFP4 uses one E4M3 scale per 16 plus an FP32 per-tensor scale (4.5 bits). The scale format decides how much of FP4's tiny grid each block actually uses.

KV CACHE & MEMORY

Foundational
The KV Cache: Bytes per Token and the Memory BudgetThe KV cache stores every past token's keys and values so decode never recomputes them. Its size per token is fixed by the attention design, and the memory left after weights and the engine's reserve decides how many sequences fit, which is the number that sets batch size, concurrency and when preemption starts.
Advanced🔒 Premium
Paged KV Memory and Block TablesPaged KV memory splits the cache into fixed-size blocks and gives each sequence a block table that maps logical positions to physical blocks, as virtual memory does with pages. It bounds waste to one partial block per sequence, which decides how many sequences fit and makes prefix sharing and copy-on-write possible.
Advanced🔒 Premium
Prefix Caching and Chained Block HashesAutomatic prefix caching keeps the KV blocks of finished requests and reuses them when a new prompt starts with the same tokens. Each full block is keyed by a hash chained from its parent block, which makes a hit mean the whole prefix matched; that decides TTFT for shared system prompts and how much cache multi-turn traffic needs.

BATCHING & SCHEDULING

Foundational
Iteration-Level Scheduling (Continuous Batching)Iteration-level scheduling, sold as continuous or in-flight batching, re-forms the batch at every forward step instead of once per batch of requests. Finished sequences leave and queued ones join on the next step, which decides how much of a GPU's decode capacity real traffic with uneven output lengths can use.
Advanced🔒 Premium
Chunked Prefill and the Token BudgetChunked prefill splits a long prompt into pieces that ride along with decode tokens, under a per-step token budget such as vLLM's max_num_batched_tokens. The budget controls how much prompt work shares a step with decode, trading prompt progress against token gaps without guaranteeing a fixed latency.
Advanced🔒 Premium
Preemption: Swap Versus RecomputeWhen the KV pool runs out mid-decode, the scheduler must evict a running sequence: either swap its cache to host memory over PCIe and copy it back later, or free it and recompute it with a fresh prefill. The cost of each depends on model size, KV bytes per token and whether prefix caching still holds the freed blocks.

DISTRIBUTED INFERENCE

Foundational
Tensor Parallelism for ServingTensor parallelism splits every weight matrix across GPUs so each reads a fraction of the model per decode step, at the price of two all-reduces per layer. This page derives where those all-reduces sit, why their cost is latency-bound in decode and bandwidth-bound in prefill, and how KV heads cap the useful TP degree.
Advanced🔒 Premium
Expert Parallelism and the MoE All-to-AllExpert parallelism places different experts on different GPUs and moves tokens to them with two all-to-alls per MoE layer. This page counts those bytes for DeepSeek-V3, shows why EP exists (it gathers enough tokens per expert to leave the bandwidth-bound regime), and simulates the load imbalance that makes the slowest GPU set the step time.
Advanced🔒 Premium
Prefill and Decode DisaggregationDisaggregated serving runs prefill and decode on separate GPU pools and ships each request's KV cache between them. This page prices the interference it removes, the transfer it adds (2.5 GiB for an 8K-token Llama 3.3 70B prompt), how to size the prefill-to-decode ratio, and when the extra hop is not worth it.

ATTENTION KERNELS

Foundational
The Online Softmax RecurrenceOnline softmax keeps a running max, a running normalizer and an unnormalized output, and rescales them when the max grows. The merge rule is exact and associative, which is what lets FlashAttention tile over keys and Flash-Decoding split one long context across thread blocks.
Advanced🔒 Premium
FlashAttention 1 to 4: What Each Generation ChangedEvery FlashAttention generation computes the same exact attention and attacks a different bottleneck: HBM traffic in FA1, poor work partitioning in FA2, Hopper's asynchronous tensor cores in FA3, and Blackwell's slow exponential units in FA4. Knowing which bottleneck each one removed tells you which kernel a given GPU should run.
Advanced🔒 Premium
Split-KV Decoding (Flash-Decoding)In decode there is one query row per sequence, so a FlashAttention kernel parallelized over batch and heads launches too few thread blocks to pull full HBM bandwidth. Split-KV divides each sequence's keys across blocks and merges the partial results exactly with their log-sum-exp, which decides whether long-context decode reaches bandwidth.

SERVING ENGINES

Foundational
The vLLM V1 Engine ArchitecturevLLM's V1 engine splits serving into an API-server process, an EngineCore process that owns the scheduler and the KV cache manager, and one worker per GPU. This page maps each component to its data structures in v0.31.0, works out the KV block pool for an 8B model on an H100, and explains why the split exists: Python overhead has to hide under a 4.8 ms decode step.
Advanced🔒 Premium
SGLang's Radix Cache and Overlap SchedulerSGLang stores every cached prefix in a radix tree keyed by token ids, so a hit can end at any token rather than at a block boundary, and its overlap scheduler prepares the next batch while the GPU runs the current one. This page walks both through the v0.5.21 source: matching, lock references, leaf-first LRU eviction, the 128-request cutoff on prefix-aware scheduling, and memory defaults.
Advanced🔒 Premium
Serving Engine Knobs and Their DefaultsA handful of engine settings decide how much KV fits, how long a step can get and how many requests run at once. This page lists the defaults in vLLM v0.31.0 and SGLang v0.5.21 as read from source, shows which knob binds first for Llama 3.3 70B on four H100s, and gives the order in which to tune them.

IMPLEMENTATION PATTERNS

Foundational
Attention With a KV Cache, From ScratchA from-scratch implementation of causal multi-head attention with grouped-query KV heads and a KV cache, in about 35 lines of numpy. Covers the offset causal mask when queries are fewer than keys, why the cache turns quadratic recompute into linear work, and the tests an interviewer runs against it.
Advanced🔒 Premium
Property-Testing Engine ComponentsHow to test serving-engine code the way graders do: state invariants, generate random operation sequences, compare against a slow reference within a tolerance derived from the number format, and shrink failures to a minimal repro. Includes a runnable stateful test for a KV block pool.
Advanced🔒 Premium
Sampling: Temperature, Top-k and Top-pHow LLM token sampling works and how to implement it correctly: temperature rescales logits, top-k keeps the k largest, top-p keeps the smallest prefix of the sorted distribution reaching probability p. Covers order of operations, ties at the cutoff, temperature 0, fully masked vocabularies and the cost at full vocabulary size.
Explore the full curriculum →

Where to apply, and official d-Matrix resources

Straight from d-Matrix: open roles and the company's own hiring guidance. Prep here, then apply there.

ApplyCareers at d-Matrix

External links to d-Matrix's own pages. Roles and processes change; always confirm on the official site.

ABOUT THE ROLE
What is an inference engineer?How inference loops are builtWhat inference roles payThe skills inference loops test
D-MATRIX INTERVIEW FAQ
Does d-Matrix hire inference engineers?▲

Yes. d-Matrix job postings seen on 16 October 2026 include Senior Staff LLM Inference Engineer, Senior Staff Inference Performance Engineer, Staff Software Engineer, SIMD Kernels. The same team posts under different titles from one req to the next, so judge a posting by its listed duties.

What is the d-Matrix inference interview process?▼
What does d-Matrix pay inference engineers?▼

Walk into your d-Matrix Inference Engineer interview ready

One answer bank covers every round, from KV budgets to kernel coding, each with the numbers worked out. Read the free answers first and decide whether the depth is what you need.

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

Unlock all answers · ₹2,000 / $25Browse free questions first

Or create a free account to unlock more free answers per topic.

Other Inference Engineer interviews to prep

Companies whose loops test the same tracks as d-Matrix's.

SnowflakeInference Engineer interview →xAI (SpaceXSI)Inference Engineer interview →NVIDIAInference Engineer interview →QualcommInference Engineer interview →AnthropicInference Engineer interview →BasetenInference Engineer interview →

Independent and not affiliated with d-Matrix. All trademarks belong to their owners.