You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: LLM/deploy-openai-llm-byoc.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,26 +1,24 @@
1
1
# Deploy OpenAI open-source models
2
2
3
-
This guide demonstrates how to deploy and perform inference using OCI Data Science Service. In this example, we will use a model downloaded from Hugging Face specifically, [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) from OpenAI.
4
-
5
-
3
+
This guide demonstrates how to deploy and perform inference using OCI Data Science Service. In this example, we will use a model downloaded from Hugging Face, specifically [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) from OpenAI.
6
4
7
5
## Required IAM Policies
8
6
9
7
Add these [policies](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/llama2#required-iam-policies) to grant access to OCI services.
10
8
11
9
## Setup
12
10
13
-
Create a data science notebook session with at least 400GB of space. We will use notebook session to -
11
+
Create a data science notebook session with at least 400GB of space. We will use a notebook session to:
14
12
1. Download model weights
15
13
2. Create Model Catalog entry
16
14
3. Deploy the model
17
15
18
-
To prepare inference container, we will use local laptop since this step requires docker commmands. The notebook session does not come with the docker tooling.
16
+
To prepare the inference container, we will use local laptop since this step requires Docker commands. The notebook session does not come with the docker tooling.
19
17
20
18
21
19
# Prepare Inference container
22
20
23
-
vLLM is an easy-to-use library for LLM inference and server. You can get the container image from [DockerHub](https://hub.docker.com/r/vllm/vllm-openai/tags).
21
+
vLLM is an easy-to-use library for LLM inference and serving. You can get the container image from [DockerHub](https://hub.docker.com/r/vllm/vllm-openai/tags).
24
22
25
23
Following commands are to run on your laptop -
26
24
@@ -41,7 +39,7 @@ You may need to docker login to the Oracle Cloud Container Registry (OCIR) first
If your tenancy is federated with Oracle Identity Cloud Service, use the format <tenancy-namespace>/oracleidentitycloudservice/<username>. You can then push the container image to the OCI Registry
42
+
If your tenancy is federated with Oracle Identity Cloud Service, use the format <tenancy-namespace>/oracleidentitycloudservice/<username>. You can then push the container image to the OCI Registry:
45
43
46
44
```shell
47
45
docker tag vllm/vllm-openai:gptoss -t <region>.ocir.io/<tenancy>/vllm-odsc/vllm-openai:gptoss
0 commit comments