Skip to content

Commit 68ed6f6

Browse files
committed
Document Sync by Tina
1 parent 2c596c6 commit 68ed6f6

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

docs/stable/store/quickstart.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,14 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
109109

110110
## Usage with vLLM
111111

112-
To use ServerlessLLM as a load format for vLLM, you need to apply our patch `serverless_llm/store/vllm_patch/sllm_load.patch` to the installed vLLM library. Therefore, please make sure you have read and followed the steps in the `vLLM Patch` section under our [installation guide](../getting_started/installation.md).
112+
:::tip
113+
To use ServerlessLLM as the load format for vLLM, you need to apply our patch `serverless_llm/store/vllm_patch/sllm_load.patch` to the installed vLLM library. Therefore, please ensure you have applied our `vLLM Patch` as instructed in [installation guide](../getting_started/installation.md).
114+
```bash
115+
VLLM_PATH=$(python -c "import vllm; import os; print(os.path.dirname(os.path.abspath(vllm.__file__)))")
116+
patch -p2 -d $VLLM_PATH < serverless_llm/store/vllm_patch/sllm_load.patch
117+
```
118+
:::
119+
113120

114121
Our api aims to be compatible with the `sharded_state` load format in vLLM. Thus, due to the model modifications about the model architecture done by vLLM, the model format for vLLM is **not** the same as we used in transformers. Thus, the `ServerlessLLM format` mentioned in the subsequent sections means the format integrated with vLLM, which is different from the `ServerlessLLM format` used in the previous sections.
115122

0 commit comments

Comments
 (0)