vLLM 0.6.6 release notes

Highlights

Support Deepseek V3 (#11523, #11502) model.
- On 8xH200s or MI300x: vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 --trust-remote-code --max-model-len 8192. The context length can be increased to about 32K beyond running into memory issue.
- For other devices, follow our distributed inference guide to enable tensor parallel and/or pipeline parallel inference
- We are just getting started for enhancing the support and unlock more performance. See #11539 for planned work.
Last mile stretch for V1 engine refactoring: API Server (#11529, #11530), penalties for sampler (#10681), prefix caching for vision language models (#11187, #11305), TP Ray executor (#11107,#11472)
Breaking change: X-Request-ID echoing is now opt-in instead of on by default for performance reason. Set --enable-request-id-headers to enable it.

Model Support

IBM Granite 3.1 (#11307), JambaForSequenceClassification model (#10860)
Add QVQ and QwQ to the list of supported models (#11509)

Performance

Cutlass 2:4 Sparsity + FP8/INT8 Quant Support (#10995)

Production Engine

Support streaming model from S3 using RunAI Model Streamer as optional loader (#10192)
Online Pooling API (#11457)
Load video from base64 (#11492)

Others

Add pypi index for every commit and nightly build (#11404)

What's Changed

[Bugfix] Set temperature=0.7 in test_guided_choice_chat by @mgoin in https://github.com/vllm-project/vllm/pull/11264
[V1] Prefix caching for vision language models by @comaniac in https://github.com/vllm-project/vllm/pull/11187
[Bugfix] Restore support for larger block sizes by @kzawora-intel in https://github.com/vllm-project/vllm/pull/11259
[Bugfix] Fix guided decoding with tokenizer mode mistral by @wallashss in https://github.com/vllm-project/vllm/pull/11046
[MISC][XPU]update ipex link for CI fix by @yma11 in https://github.com/vllm-project/vllm/pull/11278
[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support by @dsikka in https://github.com/vllm-project/vllm/pull/10995
[Bugfix] Fix broken phi3-v mm_processor_kwargs tests by @Isotr0py in https://github.com/vllm-project/vllm/pull/11263
[CI][Misc] Remove Github Action Release Workflow by @simon-mo in https://github.com/vllm-project/vllm/pull/11274
[FIX] update openai version by @jikunshang in https://github.com/vllm-project/vllm/pull/11287
[Bugfix] fix minicpmv test by @joerunde in https://github.com/vllm-project/vllm/pull/11304
[V1] VLM - enable processor cache by default by @alexm-neuralmagic in https://github.com/vllm-project/vllm/pull/11305
[Bugfix][Build/CI] Fix sparse CUTLASS compilation on CUDA [12.0, 12.2) by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/11311
[Model] IBM Granite 3.1 by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/11307
[CI] Expand test_guided_generate to test all backends by @mgoin in https://github.com/vllm-project/vllm/pull/11313
[V1] Simplify prefix caching logic by removing num_evictable_computed_blocks by @heheda12345 in https://github.com/vllm-project/vllm/pull/11310
[VLM] Merged multimodal processor for Qwen2-Audio by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11303
[Kernel] Refactor Cutlass c3x by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/10049
[Misc] Optimize ray worker initialization time by @ruisearch42 in https://github.com/vllm-project/vllm/pull/11275
[misc] benchmark_throughput : Add LoRA by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/11267
[Feature] Add load generation config from model by @liuyanyi in https://github.com/vllm-project/vllm/pull/11164
[Bugfix] Cleanup Pixtral HF code by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11333
[Model] Add JambaForSequenceClassification model by @yecohn in https://github.com/vllm-project/vllm/pull/10860
[V1] Fix multimodal profiling for Molmo by @ywang96 in https://github.com/vllm-project/vllm/pull/11325
[Model] Refactor Qwen2-VL to use merged multimodal processor by @Isotr0py in https://github.com/vllm-project/vllm/pull/11258
[Misc] Clean up and consolidate LRUCache by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11339
[Bugfix] Fix broken CPU compressed-tensors test by @Isotr0py in https://github.com/vllm-project/vllm/pull/11338
[Misc] Remove unused vllm/block.py by @Ghjk94522 in https://github.com/vllm-project/vllm/pull/11336
[CI] Adding CPU docker pipeline by @zhouyuan in https://github.com/vllm-project/vllm/pull/11261
[Bugfix][Hardware][POWERPC] Fix auto dtype failure in case of POWER10 by @Akashcodes732 in https://github.com/vllm-project/vllm/pull/11331
[ci][gh200] dockerfile clean up by @youkaichao in https://github.com/vllm-project/vllm/pull/11351
[Misc] Add tqdm progress bar during graph capture by @mgoin in https://github.com/vllm-project/vllm/pull/11349
[Bugfix] Fix spec decoding when seed is none in a batch by @wallashss in https://github.com/vllm-project/vllm/pull/10863
[misc] add early error message for custom ops by @youkaichao in https://github.com/vllm-project/vllm/pull/11355
[doc] backward compatibility for 0.6.4 by @youkaichao in https://github.com/vllm-project/vllm/pull/11359
[V1] Fix profiling for models with merged input processor by @ywang96 in https://github.com/vllm-project/vllm/pull/11370
[CI/Build] fix pre-compiled wheel install for exact tag by @dtrifiro in https://github.com/vllm-project/vllm/pull/11373
[Core] Loading model from S3 using RunAI Model Streamer as optional loader by @omer-dayan in https://github.com/vllm-project/vllm/pull/10192
[Bugfix] Don't log OpenAI field aliases as ignored by @mgoin in https://github.com/vllm-project/vllm/pull/11378
[doc] explain nccl requirements for rlhf by @youkaichao in https://github.com/vllm-project/vllm/pull/11381
Add ray[default] to wget to run distributed inference out of box by @Jeffwan in https://github.com/vllm-project/vllm/pull/11265
[V1][Bugfix] Skip hashing empty or None mm_data by @WoosukKwon in https://github.com/vllm-project/vllm/pull/11386
[Bugfix] update should_ignore_layer by @horheynm in https://github.com/vllm-project/vllm/pull/11354
[V1] Make AsyncLLMEngine v1-v0 opaque by @rickyyx in https://github.com/vllm-project/vllm/pull/11383
[Bugfix] Fix issues for Pixtral-Large-Instruct-2411 by @ywang96 in https://github.com/vllm-project/vllm/pull/11393
[CI] Fix flaky entrypoint tests by @ywang96 in https://github.com/vllm-project/vllm/pull/11403
[cd][release] add pypi index for every commit and nightly build by @youkaichao in https://github.com/vllm-project/vllm/pull/11404
[cd][release] fix race conditions by @youkaichao in https://github.com/vllm-project/vllm/pull/11407
[Bugfix] Fix fully sharded LoRAs with Mixtral by @n1hility in https://github.com/vllm-project/vllm/pull/11390
[CI] Unboock H100 Benchmark by @simon-mo in https://github.com/vllm-project/vllm/pull/11419
[misc][perf] remove old code by @youkaichao in https://github.com/vllm-project/vllm/pull/11425
mypy type checking for vllm/worker by @lucas-tucker in https://github.com/vllm-project/vllm/pull/11418
[Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF by @mgoin in https://github.com/vllm-project/vllm/pull/11389
[Bugfix] torch nightly version in ROCm installation guide by @terrytangyuan in https://github.com/vllm-project/vllm/pull/11423
[Misc] Add assertion and helpful message for marlin24 compressed models by @dsikka in https://github.com/vllm-project/vllm/pull/11388
[Misc] add w8a8 asym models by @dsikka in https://github.com/vllm-project/vllm/pull/11075
[CI] Expand OpenAI test_chat.py guided decoding tests by @mgoin in https://github.com/vllm-project/vllm/pull/11048
[Bugfix] Add kv cache scales to gemma2.py by @mgoin in https://github.com/vllm-project/vllm/pull/11269
[Doc] Fix typo in the help message of '--guided-decoding-backend' by @yansh97 in https://github.com/vllm-project/vllm/pull/11440
[Docs] Convert rST to MyST (Markdown) by @rafvasq in https://github.com/vllm-project/vllm/pull/11145
[V1] TP Ray executor by @ruisearch42 in https://github.com/vllm-project/vllm/pull/11107
[Misc]Suppress irrelevant exception stack trace information when CUDA… by @shiquan1988 in https://github.com/vllm-project/vllm/pull/11438
[Frontend] Online Pooling API by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11457
[Bugfix] Fix Qwen2-VL LoRA weight loading by @jeejeelee in https://github.com/vllm-project/vllm/pull/11430
[Bugfix][Hardware][CPU] Fix CPU input_positions creation for text-only inputs with mrope by @Isotr0py in https://github.com/vllm-project/vllm/pull/11434
[OpenVINO] Fixed installation conflicts by @ilya-lavrenov in https://github.com/vllm-project/vllm/pull/11458
[attn][tiny fix] fix attn backend in MultiHeadAttention by @MengqingCao in https://github.com/vllm-project/vllm/pull/11463
[Misc] Move weights mapper by @jeejeelee in https://github.com/vllm-project/vllm/pull/11443
[Bugfix] Fix issues in CPU build Dockerfile. Fixes #9182 by @terrytangyuan in https://github.com/vllm-project/vllm/pull/11435
[Model] Automatic conversion of classification and reward models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11469
[V1] Unify VLLM_ENABLE_V1_MULTIPROCESSING handling in RayExecutor by @ruisearch42 in https://github.com/vllm-project/vllm/pull/11472
[Misc] Update disaggregation benchmark scripts and test logs by @Jeffwan in https://github.com/vllm-project/vllm/pull/11456
[Frontend] Enable decord to load video from base64 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11492
[Doc] Improve GitHub links by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11491
[Misc] Move some multimodal utils to modality-specific modules by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11494
Mypy checking for vllm/compilation by @lucas-tucker in https://github.com/vllm-project/vllm/pull/11496
[Misc][LoRA] Fix LoRA weight mapper by @jeejeelee in https://github.com/vllm-project/vllm/pull/11495
[Doc] Add QVQ and QwQ to the list of supported models by @ywang96 in https://github.com/vllm-project/vllm/pull/11509
[V1] Adding min tokens/repetition/presence/frequence penalties to V1 sampler by @sroy745 in https://github.com/vllm-project/vllm/pull/10681
[Model] Modify MolmoForCausalLM MLP by @jeejeelee in https://github.com/vllm-project/vllm/pull/11510
[Misc] Add placeholder module by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/11501
[Doc] Add video example to openai client for multimodal by @Isotr0py in https://github.com/vllm-project/vllm/pull/11521
[V1] [1/N] [Breaking Change] API Server (Remove Proxy) by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/11529
[Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization by @mgoin in https://github.com/vllm-project/vllm/pull/11523
[2/N] API Server: Avoid ulimit footgun by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/11530
Deepseek v3 by @simon-mo in https://github.com/vllm-project/vllm/pull/11502

Full Changelog: https://github.com/vllm-project/vllm/compare/v0.6.5...v0.6.6