vLLM 0.8.2 release notes

This release contains important bug fix for the V1 engine's memory usage. We highly recommend you upgrading!

Highlights

Revert "Use uv python for docker rather than ppa:deadsnakess/ppa (#13569)" (#15377)
Remove openvino support in favor of external plugin (#15339)

V1 Engine

Fix V1 Engine crash while handling requests with duplicate request id (#15043)
Support FP8 KV Cache (#14570, #15191)
Add flag to disable cascade attention (#15243)
Scheduler Refactoring: Add Scheduler Interface (#15250)
Structured Output
- Add disable-any-whitespace option support for xgrammar (#15316)
- guidance backend for structured output + auto fallback mode (#14779)
Spec Decode
- Enable spec decode for top-p & top-k sampling (#15063)
- Use better defaults for N-gram (#15358)
- Update target_logits in place for rejection sampling (#15427)
AMD
- Enable Triton(ROCm) Attention backend for Nvidia GPUs (#14071)
TPU
- Support V1 Sampler for ragged attention (#14227)
- Tensor parallel MP support (#15059)
- MHA Pallas backend (#15288)

Features

Integrate fastsafetensors loader for loading model weights (#10647)
Add guidance backend for structured output (#14589)

Others

Add Kubernetes deployment guide with CPUs (#14865)
Support reset prefix cache by specified device (#15003)
Support tool calling and reasoning parser (#14511)
Support --disable-uvicorn-access-log parameters (#14754)
Support Tele-FLM Model (#15023)
Add pipeline parallel support to TransformersModel (#12832)
Enable CUDA graph support for llama 3.2 vision (#14917)

What's Changed

[FEAT]Support reset prefix cache by specified device by @maobaolong in https://github.com/vllm-project/vllm/pull/15003
[BugFix][V1] Update stats.py by @WrRan in https://github.com/vllm-project/vllm/pull/15139
[V1][TPU] Change kv cache shape. by @vanbasten23 in https://github.com/vllm-project/vllm/pull/15145
[FrontEnd][Perf] merge_async_iterators fast-path for single-prompt requests by @njhill in https://github.com/vllm-project/vllm/pull/15150
[Docs] Annouce Ollama and Singapore Meetups by @simon-mo in https://github.com/vllm-project/vllm/pull/15161
[V1] TPU - Tensor parallel MP support by @alexm-redhat in https://github.com/vllm-project/vllm/pull/15059
[BugFix] Lazily import XgrammarBackend to avoid early cuda init by @njhill in https://github.com/vllm-project/vllm/pull/15171
[Doc] Clarify run vllm only on one node in distributed inference by @ruisearch42 in https://github.com/vllm-project/vllm/pull/15148
Fix broken tests by @jovsa in https://github.com/vllm-project/vllm/pull/14713
[Bugfix] Fix embedding assignment for InternVL-based models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15086
fix "Total generated tokens:" is 0 if using --backend tgi and --endpo… by @sywangyi in https://github.com/vllm-project/vllm/pull/14673
[V1][TPU] Support V1 Sampler for ragged attention by @NickLucche in https://github.com/vllm-project/vllm/pull/14227
[Benchmark] Allow oversample request in benchmark dataset by @JenZhao in https://github.com/vllm-project/vllm/pull/15170
[Core][V0] Add guidance backend for structured output by @russellb in https://github.com/vllm-project/vllm/pull/14589
[Doc] Update Mistral Small 3.1/Pixtral example by @ywang96 in https://github.com/vllm-project/vllm/pull/15184
[Misc] support --disable-uvicorn-access-log parameters by @chaunceyjiang in https://github.com/vllm-project/vllm/pull/14754
[Attention] Flash Attention 3 - fp8 by @mickaelseznec in https://github.com/vllm-project/vllm/pull/14570
[Doc] Update README.md by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15187
Enable CUDA graph support for llama 3.2 vision by @mritterfigma in https://github.com/vllm-project/vllm/pull/14917
typo: Update config.py by @WrRan in https://github.com/vllm-project/vllm/pull/15189
[Frontend][Bugfix] support prefill decode disaggregation on deepseek by @billishyahao in https://github.com/vllm-project/vllm/pull/14824
[release] Tag vllm-cpu with latest upon new version released by @khluu in https://github.com/vllm-project/vllm/pull/15193
Fixing Imprecise Type Annotations by @WrRan in https://github.com/vllm-project/vllm/pull/15192
[macOS] Ugrade pytorch to 2.6.0 by @linktohack in https://github.com/vllm-project/vllm/pull/15129
[Bugfix] Multi-video inference on LLaVA-Onevision by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15082
Add user forum to README by @hmellor in https://github.com/vllm-project/vllm/pull/15220
Fix env vars for running Ray distributed backend on GKE by @richardsliu in https://github.com/vllm-project/vllm/pull/15166
Replace misc issues with link to forum by @hmellor in https://github.com/vllm-project/vllm/pull/15226
[ci] feat: make the test_torchrun_example run with tp=2, external_dp=2 by @vermouth1992 in https://github.com/vllm-project/vllm/pull/15172
[Bugfix] fix V1 Engine crash while handling requests with duplicate request id by @JasonJ2021 in https://github.com/vllm-project/vllm/pull/15043
[V1] Add flag to disable cascade attention by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15243
Enforce that TP > 1 is not supported for Mamba2 if Quantization is Enabled. by @fabianlim in https://github.com/vllm-project/vllm/pull/14617
[V1] Scheduler Refactoring [1/N] - Add Scheduler Interface by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15250
[CI/Build] LoRA : make add_lora_test safer by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/15181
Fix CUDA kernel index data type in vllm/csrc/quantization/fused_kernels/layernorm_utils.cuh +10 by @houseroad in https://github.com/vllm-project/vllm/pull/15159
[Misc] Clean up the BitsAndBytes arguments by @jeejeelee in https://github.com/vllm-project/vllm/pull/15140
[ROCM] Upgrade torch to 2.6 by @SageMoore in https://github.com/vllm-project/vllm/pull/15244
[Bugfix] Fix incorrect qwen2.5-vl attention mask pre-computation by @Isotr0py in https://github.com/vllm-project/vllm/pull/15200
Mention extra_body as a way top pass vLLM only parameters using the OpenAI client by @hmellor in https://github.com/vllm-project/vllm/pull/15240
[V1][TPU] Speed up top-k on TPU by using torch.topk by @hyeygit in https://github.com/vllm-project/vllm/pull/15242
[Bugfix] detect alibi and revert to FA2 by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/15231
[Model] RE: Mamba2 Prefill Performance Tweaks: Fixing Flurry of Unnecessary Memory Copies by @cyang49 in https://github.com/vllm-project/vllm/pull/14857
[Docs] Trim the latest news in README by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15261
[Misc] Better RayExecutor and multiprocessing compatibility by @comaniac in https://github.com/vllm-project/vllm/pull/14705
Add an example for reproducibility by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15262
[Hardware][TPU] Add check for no additional graph compilation during runtime by @lsy323 in https://github.com/vllm-project/vllm/pull/14710
[V1] Enable Triton(ROCm) Attention backend for Nvidia GPUs by @Isotr0py in https://github.com/vllm-project/vllm/pull/14071
[Doc] Update LWS docs by @Edwinhr716 in https://github.com/vllm-project/vllm/pull/15163
[V1] Avoid redundant input processing in n>1 case by @njhill in https://github.com/vllm-project/vllm/pull/14985
[Feature] specify model in config.yaml by @wayzeng in https://github.com/vllm-project/vllm/pull/14855
[Bugfix] Add int8 torch dtype for KVCache by @shen-shanshan in https://github.com/vllm-project/vllm/pull/15260
[Misc] Add attention mask pre-computation optimization back to Qwen2.5-VL by @Isotr0py in https://github.com/vllm-project/vllm/pull/15273
[Bugfix] Fix incorrect resolving order for transformers fallback by @Isotr0py in https://github.com/vllm-project/vllm/pull/15279
[V1] Fix wrong import path of get_flash_attn_version by @lhtin in https://github.com/vllm-project/vllm/pull/15280
[Bugfix] Fix broken kernel test due to missing rename for v1 Triton backend by @Isotr0py in https://github.com/vllm-project/vllm/pull/15282
[Misc] Add cProfile helpers by @russellb in https://github.com/vllm-project/vllm/pull/15074
[v1] Refactor KVCacheConfig by @heheda12345 in https://github.com/vllm-project/vllm/pull/14079
[Bugfix][VLM] fix llava processor by @MengqingCao in https://github.com/vllm-project/vllm/pull/15285
Revert "[Feature] specify model in config.yaml (#14855)" by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15293
[TPU][V1] MHA Pallas backend by @NickLucche in https://github.com/vllm-project/vllm/pull/15288
[Build/CI] Fix env var typo by @russellb in https://github.com/vllm-project/vllm/pull/15305
[Misc] Increase RayDistributedExecutor RAY_CGRAPH_get_timeout by @ruisearch42 in https://github.com/vllm-project/vllm/pull/15301
[Bugfix][V0] Multi-sequence logprobs streaming edge case by @andylolu2 in https://github.com/vllm-project/vllm/pull/15259
[FEAT] [ROCm]: Add AITER RMS Norm (Layer Norm) Feature by @tjtanaa in https://github.com/vllm-project/vllm/pull/14959
[Doc] add load_format items in docs by @wwl2755 in https://github.com/vllm-project/vllm/pull/14804
[Bugfix] Fix torch.compile raise FileNotFoundError by @jeejeelee in https://github.com/vllm-project/vllm/pull/15278
[Bugfix] LoRA V0 - Fix case where max_num_seqs is between cudagraph capture sizes by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/15308
[Model] Support Tele-FLM Model by @atone in https://github.com/vllm-project/vllm/pull/15023
[V1] Add disable-any-whitespace option support for xgrammar by @russellb in https://github.com/vllm-project/vllm/pull/15316
[BugFix][Typing] Fix Imprecise Type Annotations by @WrRan in https://github.com/vllm-project/vllm/pull/15208
Remove openvino support in favor of external plugin by @russellb in https://github.com/vllm-project/vllm/pull/15339
[doc] Add back previous news by @heheda12345 in https://github.com/vllm-project/vllm/pull/15331
Fix v1 supported oracle for worker-cls and worker-extension-cls by @hijkzzz in https://github.com/vllm-project/vllm/pull/15324
[V1][Usage] Refactor speculative decoding configuration and tests by @ShangmingCai in https://github.com/vllm-project/vllm/pull/14434
[ci/build] update torch nightly version for GH200 by @youkaichao in https://github.com/vllm-project/vllm/pull/15135
[ci/build] fix broken tests in LLM.collective_rpc by @youkaichao in https://github.com/vllm-project/vllm/pull/15350
[Misc] Add tuned R1 w8a8 and MoE configs for NVIDIA L20 by @DefTruth in https://github.com/vllm-project/vllm/pull/15322
[Bugfix] fix torch.compiled cache hash error by @DefTruth in https://github.com/vllm-project/vllm/pull/14953
[V1][Spec Decode] Respect prompt_lookup_max by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15348
[V1][Spec Decode] Use better defaults for N-gram by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15358
[Frontend] Support tool calling and reasoning parser by @WangErXiao in https://github.com/vllm-project/vllm/pull/14511
[Misc][Doc] Add note regarding loading generation_config by default by @ywang96 in https://github.com/vllm-project/vllm/pull/15281
[V1] Enable V1 Fp8 cache for FA3 in the oracle by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/15191
[Fix] [torch.compile] Improve UUID system for custom passes by @ProExpertProg in https://github.com/vllm-project/vllm/pull/15249
Fix non-contiguous input passed to Marlin kernel by @Qubitium in https://github.com/vllm-project/vllm/pull/15319
[Misc] Upgrade BNB version by @jeejeelee in https://github.com/vllm-project/vllm/pull/15183
[Misc] Remove ignore_reinit_error for ray.init() by @ruisearch42 in https://github.com/vllm-project/vllm/pull/15373
[Bugfix][V1] Avoid importing PreTrainedModel by @HollowMan6 in https://github.com/vllm-project/vllm/pull/15366
[Misc] Update guided decoding logs to debug by @sfbemerk in https://github.com/vllm-project/vllm/pull/15310
Revert "[CI/Build] Use uv python for docker rather than ppa:deadsnakess/ppa (#13569)" by @simon-mo in https://github.com/vllm-project/vllm/pull/15377
[Kernel] allow non-contiguous input for marlin kernel by @jinzhen-lin in https://github.com/vllm-project/vllm/pull/14658
Fix zmq IPv6 URL format error by @russellb in https://github.com/vllm-project/vllm/pull/15341
[Bugfix] Fix chat template loading by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15143
[distributed] fix dp group by @youkaichao in https://github.com/vllm-project/vllm/pull/15355
[Core] Integrate fastsafetensors loader for loading model weights by @manish-sethi in https://github.com/vllm-project/vllm/pull/10647
[Core] Don't force uppercase for VLLM_LOGGING_LEVEL by @russellb in https://github.com/vllm-project/vllm/pull/15306
[V1][Minor] fix comments by @Chen-0210 in https://github.com/vllm-project/vllm/pull/15392
[MISC] Refine no available block debug msg by @yiliu30 in https://github.com/vllm-project/vllm/pull/15076
[V1] Aggregate chunked prompt logprobs in model runner by @njhill in https://github.com/vllm-project/vllm/pull/14875
[Hardware][Gaudi][Feature] Enable Dynamic MoE for Mixtral by @zhenwei-intel in https://github.com/vllm-project/vllm/pull/12303
[DOC] Add Kubernetes deployment guide with CPUs by @terrytangyuan in https://github.com/vllm-project/vllm/pull/14865
[Doc] Update docs on handling OOM by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15357
[V1][Perf] Simpler request output queues by @njhill in https://github.com/vllm-project/vllm/pull/15156
[BugFix][V1] Quick fix for min_tokens with multiple EOS by @njhill in https://github.com/vllm-project/vllm/pull/15407
[Hardware][TPU] Skip failed compilation test by @lsy323 in https://github.com/vllm-project/vllm/pull/15421
[Build] Cython compilation support fix by @gshtras in https://github.com/vllm-project/vllm/pull/14296
[ROCm][Kernel] MoE weights padding by @gshtras in https://github.com/vllm-project/vllm/pull/14454
[V1][Spec Decode] Enable spec decode for top-p & top-k sampling by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15063
[Minor][Spec Decode] Remove compiled_softmax by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15416
Add pipeline parallel support to TransformersModel by @hmellor in https://github.com/vllm-project/vllm/pull/12832
[Misc] Remove LoRA log by @jeejeelee in https://github.com/vllm-project/vllm/pull/15388
Revert "Fix non-contiguous input passed to Marlin kernel (#15319)" by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/15398
[Bugfix] Fixed the issue of not being able to input video and image simultaneously by @chaunceyjiang in https://github.com/vllm-project/vllm/pull/15387
[V1] guidance backend for structured output + auto fallback mode by @russellb in https://github.com/vllm-project/vllm/pull/14779
[V1][Spec Decode] Update target_logits in place for rejection sampling by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15427

Full Changelog: https://github.com/vllm-project/vllm/compare/v0.8.1...v0.8.2