0.6.3.post1

Highlights

New Models

• Support Ministral 3B and Ministral 8B via interleaved attention (#9414)
• Support multiple and interleaved images for Llama3.2 (#9095)
• Support VLM2Vec, the first multimodal embedding model in vLLM (#9303)

Important bug fix

• Fix chat API continuous usage stats (#9357)
• Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids (#9034)
• Fix Molmo text-only input bug (#9397)
• Fix CUDA 11.8 Build (#9386)
• Fix _version.py not found issue (#9375)

Other Enhancements

• Remove block manager v1 and make block manager v2 default (#8704)
• Spec Decode Optimize ngram lookup performance (#9333)

What's Changed

• [TPU] Fix TPU SMEM OOM by Pallas paged attention kernel by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9350
• [Frontend] merge beam search implementations by @LunrEclipse in https://github.com/vllm-project/vllm/pull/9296
• [Model] Make llama3.2 support multiple and interleaved images by @xiangxu-google in https://github.com/vllm-project/vllm/pull/9095
• [Bugfix] Clean up some cruft in mamba.py by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9343
• [Frontend] Clarify model_type error messages by @stevegrubb in https://github.com/vllm-project/vllm/pull/9345
• [Doc] Fix code formatting in spec_decode.rst by @mgoin in https://github.com/vllm-project/vllm/pull/9348
• [Bugfix] Update InternVL input mapper to support image embeds by @hhzhang16 in https://github.com/vllm-project/vllm/pull/9351
• [BugFix] Fix chat API continuous usage stats by @njhill in https://github.com/vllm-project/vllm/pull/9357
• pass ignore_eos parameter to all benchmark_serving calls by @gracehonv in https://github.com/vllm-project/vllm/pull/9349
• [Misc] Directly use compressed-tensors for checkpoint definitions by @mgoin in https://github.com/vllm-project/vllm/pull/8909
• [Bugfix] Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids by @CatherineSue in https://github.com/vllm-project/vllm/pull/9034
• [Bugfix][CI/Build] Fix CUDA 11.8 Build by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9386
• [Bugfix] Molmo text-only input bug fix by @mrsalehi in https://github.com/vllm-project/vllm/pull/9397
• [Misc] Standardize RoPE handling for Qwen2-VL by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9250
• [Model] VLM2Vec, the first multimodal embedding model in vLLM by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9303
• [CI/Build] Test VLM embeddings by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9406
• [Core] Rename input data types by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8688
• [Misc] Consolidate example usage of OpenAI client for multimodal models by @ywang96 in https://github.com/vllm-project/vllm/pull/9412
• [Model] Support SDPA attention for Molmo vision backbone by @Isotr0py in https://github.com/vllm-project/vllm/pull/9410
• Support mistral interleaved attn by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/9414
• [Kernel][Model] Improve continuous batching for Jamba and Mamba by @mzusman in https://github.com/vllm-project/vllm/pull/9189
• [Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft by @streaver91 in https://github.com/vllm-project/vllm/pull/9396
• [Performance][Spec Decode] Optimize ngram lookup performance by @LiuXiaoxuanPKU in https://github.com/vllm-project/vllm/pull/9333
• [CI/Build] mypy: Resolve some errors from checking vllm/engine by @russellb in https://github.com/vllm-project/vllm/pull/9267
• [Bugfix][Kernel] Prevent integer overflow in fp8 dynamic per-token quantize kernel by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9425
• [BugFix] [Kernel] Fix GPU SEGV occurring in int8 kernels by @rasmith in https://github.com/vllm-project/vllm/pull/9391
• Add notes on the use of Slack by @terrytangyuan in https://github.com/vllm-project/vllm/pull/9442
• [Kernel] Add Exllama as a backend for compressed-tensors by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9395
• [Misc] Print stack trace using logger.exception by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9461
• [misc] CUDA Time Layerwise Profiler by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/8337
• [Bugfix] Allow prefill of assistant response when using mistral_common by @sasha0552 in https://github.com/vllm-project/vllm/pull/9446
• [TPU] Call torch._sync(param) during weight loading by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9437
• [Hardware][CPU] compressed-tensor INT8 W8A8 AZP support by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/9344
• [Core] Deprecating block manager v1 and make block manager v2 default by @KuntaiDu in https://github.com/vllm-project/vllm/pull/8704
• [CI/Build] remove .github from .dockerignore, add dirty repo check by @dtrifiro in https://github.com/vllm-project/vllm/pull/9375

Full Changelog: https://github.com/vllm-project/vllm/compare/v0.6.3...v0.6.3.post1