vllm-omni post init and roadmap updates

Author: hsliuustc0106

_posts/2025-11-30-vllm-omni.md

@@ -0,0 +1,109 @@
+## **Announcing vLLM-Omni: Easy, Fast, and Cheap Omni-Modality Model Serving" author: "The vLLM-Omni Team"**
+
+We are excited to announce the official release of **vLLM-Omni**, a major extension of the vLLM ecosystem designed to support the next generation of AI: omni-modality models.
+
+Since its inception, vLLM has focused on high-throughput, memory-efficient serving for Large Language Models (LLMs). However, the landscape of generative AI is shifting rapidly. Models are no longer just about text-in, text-out. Today's state-of-the-art models reason across text, images, audio, and video, and they generate heterogeneous outputs using diverse architectures.
+
+**vLLM-Omni** answers this call, extending vLLM’s legendary performance to the world of multi-modal and non-autoregressive inference.
+
+\<p align="center"\>  
+\<img src="/assets/figures/vllm-omni-logo-text-dark.png" alt="vLLM Omni Logo" width="60%"\>  
+\</p\>
+
+## **Why vLLM-Omni?**
+
+Traditional serving engines were optimized for text-based Autoregressive (AR) tasks. As models evolve into "omni" agents—capable of seeing, hearing, and speaking—the serving infrastructure must evolve with them.
+
+vLLM-Omni addresses three critical shifts in model architecture:
+
+1. **True Omni-Modality:** Processing and generating Text, Image, Video, and Audio seamlessly.  
+2. **Beyond Autoregression:** Extending vLLM's efficient memory management to **Diffusion Transformers (DiT)** and other parallel generation models.  
+3. **Heterogeneous Pipelines:** Managing complex workflows where a single request might trigger a visual encoder, an AR reasoning step, and a diffusion-based video generation step.
+
+## **Inside the Architecture**
+
+vLLM-Omni is not just a wrapper; it is a re-imagining of how vLLM handles data flow. It introduces a fully disaggregated pipeline that allows for dynamic resource allocation across different stages of generation.
+
+\<p align="center"\>  
+\<img src="/assets/figures/omni-modality-model-architecture.png" alt="Omni-modality model architecture" width="80%"\>  
+\</p\>  
+As shown above, the architecture unifies distinct phases:
+
+* **Modality Encoders:** Efficiently processing inputs (ViT, T5, etc.)  
+* **LLM Core:** leveraging vLLM's PagedAttention for the autoregressive reasoning stage.  
+* **Modality Generators:** High-performance serving for DiT and other decoding heads to produce rich media outputs.
+
+### **Key Features**
+
+* **Simplicity:** If you know how to use vLLM, you know how to use vLLM-Omni. We maintain seamless integration with Hugging Face models and offer an OpenAI-compatible API server.  
+
+# todo @liuhongsheng, add the vLLM-Omni architecture
+
+
+* **Flexibility:** With the OmniStage abstraction, we provide a simple and straightforward way to support various Omni-Modality models including Qwen-Omni, Qwen-Image, SD models.
+
+
+* **Performance:** We utilize pipelined stage execution to overlap computation, ensuring that while one stage is processing, others aren't idle. 
+
+# todo @zhoutaichang, please add a figure to illustrate the pipelined stage execution.
+
+## **Performance**
+
+We benchmarked vLLM-Omni against Hugging Face Transformers to demonstrate the efficiency gains in omni-modal serving.
+
+| Metric | vLLM-Omni | HF Transformers | Improvement |
+| :---- | :---- | :---- | :---- |
+| **Throughput** (req/s) | **TBD** | TBD | **TBD x** |
+| **Latency** (TTFT ms) | **TBD** | TBD | **TBD x** |
+| **GPU Memory** (GB) | **TBD** | TBD | **TBD %** |
+
+*Note: Benchmarks were run on \[Insert Hardware Specs\] using \[Insert Model Name\].*
+
+## **Future Roadmap**
+
+vLLM-Omni is evolving rapidly. Our roadmap is focused on expanding model support and pushing the boundaries of efficient inference even further.
+
+* **Expanded Model Support:** We plan to support a wider range of open-source omni-models and diffusion transformers as they emerge.  
+* **Deeper vLLM Integration:** merging core omni-features upstream to make multi-modality a first-class citizen in the entire vLLM ecosystem.
+* **Diffusion Acceleration:** parallel inference(DP/TP/SP/USP...),  cache acceleration(TeaCache/DBCache...) and compute acceleration(quantization/sparse attn...).
+* **Full disaggregation:** Based on the OmniStage abstraction, we expect to support full disaggregation (encoder/prefill/decode/generation) across different inference stages in order to improve throughput and reduce latency.
+* **Hardware Support:** Following the hardware plugin system, we plan to expand our support for various hardware backends to ensure vLLM-Omni runs efficiently everywhere.  
+
+Contributions and collabrations from the open source community are welcome.
+
+## **Getting Started**
+
+Getting started with vLLM-Omni is straightforward. The initial release is built on top of vLLM v0.11.0.
+
+### **Installation**
+
+First, set up your environment:
+
+\# Create a virtual environment  
+uv venv \--python 3.12 \--seed  
+source .venv/bin/activate
+
+\# Install the base vLLM  
+uv pip install vllm==0.11.0 \--torch-backend=auto
+
+Next, install the vLLM-Omni extension:
+
+git clone \[https://github.com/vllm-project/vllm-omni.git\](https://github.com/vllm-project/vllm-omni.git)  
+cd vllm\_omni  
+uv pip install \-e .
+
+### **Running the Qwen3-Omni model**
+
+@huayongxiang, add the gradio example for Qwen3-Omni model inference
+
+Check out our [examples directory](https://www.google.com/search?q=https://github.com/vllm-project/vllm-omni/tree/main/examples) for specific scripts to launch image, audio, and video generation workflows.
+
+## **Join the Community**
+
+This is just the beginning for omni-modality serving. We are actively developing support for more architectures and invite the community to help shape the future of vLLM-Omni.
+
+@gaohan, update the links after vllm-omni released
+* **Code & Docs:** [GitHub Repository](https://github.com/vllm-project/vllm-omni) | [Documentation](https://vllm-omni.readthedocs.io/en/latest/)  
+* **Weekly Meeting:** Join us every Wednesday at 11:30 (UTC+8) to discuss roadmap and features. [Join here](https://tinyurl.com/vllm-omni-meeting).
+
+Let's build the future of omni-modal serving together\!