The current bottleneck in large language model inference is rarely raw compute power, but rather the agonizingly slow process of generating tokens one by one. For developers deploying massive models on AMD Instinct hardware, the challenge has been bridging the gap between theoretical TFLOPS and actual tokens per second. This week, the focus in the inference community has shifted toward speculative decoding, a technique that attempts to predict multiple future tokens and verify them in a single pass, effectively turning a sequential chore into a parallel operation.

The Architecture of vLLM Speculative Support on ROCm

vLLM has officially extended its speculative decoding capabilities to the AMD Instinct MI300X and MI355X GPUs, leveraging the ROCm software platform. The implementation centers on a draft-and-verify mechanism where a lightweight draft model proposes a sequence of candidate tokens, which the larger target model then validates in a single forward pass. To provide flexibility across different model architectures, vLLM now supports five distinct drafting methodologies: Native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark.

Configuration is handled via the `--speculative-config` argument. For Native MTP, the target model already contains the necessary auxiliary prediction components, meaning no external model path is required. However, for the other four methods, users must specify the path to a trained draft checkpoint within the model field.

bash

Native MTP 설정 예시

--speculative-config "{"method": "mtp", "num_speculative_tokens": 5}"

별도 드래프트 모델(EAGLE-3 등) 설정 예시

--speculative-config "{"method": "eagle3", "model": "checkpoint_path", "num_speculative_tokens": 5}"

These draft weights are sourced from industry leaders and research labs, including Google for Gemma 4 MTP, Z-Lab for DFlash, and Red Hat AI for EAGLE-3, DFlash, and DSpark, as well as DeepSeek. Because every method except Native MTP requires loading additional weights into the GPU, practitioners must ensure sufficient VRAM headroom to avoid out-of-memory errors during the loading phase.

Parallelism versus Sequence: The Drafting Divergence

The real technical tension lies in how these five methods generate their candidates. The first group, consisting of Native MTP, Gemma 4 MTP, and EAGLE-3, utilizes an autoregressive approach. Native MTP predicts the next candidate by combining the target model's hidden representations with the current token. EAGLE-3 takes this further by fusing hidden states extracted from three distinct stages of the target transformer—the beginning, middle, and end—to feed the draft decoder.

In contrast, DFlash and DSpark move away from sequential generation in favor of parallel prediction. DFlash identifies an anchor token confirmed by the target model and then predicts all subsequent masked positions simultaneously in one forward pass. While faster, this parallel approach introduces a risk of semantic incoherence. For instance, a parallel model might predict "of course" and "no problem" simultaneously, accidentally blending them into a nonsensical phrase like "of problem."

To solve this, DSpark introduces a lightweight sequential Markov Head behind the DFlash parallel backbone. This head applies a bias based on previously selected tokens to correct inconsistencies, ensuring that the parallel speed does not come at the cost of linguistic logic. Regardless of the drafting method, the verification phase remains a strict left-to-right process. The target model evaluates the proposed sequence and accepts tokens only until the first rejection occurs. Every token following the first rejected candidate is discarded, and the target model generates a corrected token to restart the next round of speculation.

Benchmarking Throughput and the Cost of Speculation

Performance data from the AMD MI300X and MI355X reveals that the gains from speculative decoding are highly dependent on the specific model and dataset pairing. For the gemma-4-26B-A4B-it model, DFlash achieved a 2.87x throughput increase on the MATH500 dataset, while Gemma 4 MTP saw a 2.74x boost on GSM8K. The Kimi-K2.5 model also showed significant acceleration, reaching up to 2.68x throughput when paired with DFlash. In the case of Qwen3.5-122B-A10B, Native MTP delivered a 2.20x efficiency gain on MATH500.

However, the data also warns that speculative decoding is not a universal win. When applying EAGLE-3 to the Qwen3-8B model, performance exceeded the baseline on GSM8K and HumanEval but actually fell below the baseline on MATH500. This discrepancy highlights a critical reality: the effectiveness of the system depends entirely on the draft model's accuracy and the resulting acceptance rate of the target model.

The primary tuning knob for operators is `num_speculative_tokens` (N). In autoregressive methods, increasing N typically leads to a gradual rise in throughput that eventually plateaus. DFlash and DSpark often peak at N=7. Pushing N too high creates a point of diminishing returns where the computational cost of generating and verifying low-probability tokens outweighs the benefits of the few that are actually accepted.

To optimize these pipelines, engineers must monitor two specific metrics: `mean_accepted_length` and `per_position_acceptance_rate`. If the per-position acceptance rate drops sharply after a certain index, it indicates that the draft model is guessing blindly beyond that point. In such cases, reducing the `num_speculative_tokens` value is the only way to eliminate wasted compute and restore overall system throughput.

This integration marks a significant step in making non-Nvidia hardware a first-class citizen for high-performance LLM serving.