|
| 1 | +# Oracle OCI Generative AI Tracing with Openlayer |
| 2 | + |
| 3 | +This directory contains examples for integrating Oracle Cloud Infrastructure (OCI) Generative AI with Openlayer tracing. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +Oracle OCI Generative AI is a fully managed service that provides state-of-the-art, customizable large language models (LLMs) through a single API. The Openlayer integration allows you to automatically trace and monitor all interactions with OCI Generative AI models. |
| 8 | + |
| 9 | +## Prerequisites |
| 10 | + |
| 11 | +1. **OCI Account**: Access to Oracle Cloud Infrastructure with Generative AI service enabled |
| 12 | +2. **OCI Configuration**: Properly configured OCI CLI or config file |
| 13 | +3. **Python Packages**: |
| 14 | + ```bash |
| 15 | + pip install oci openlayer |
| 16 | + ``` |
| 17 | + |
| 18 | +## Files |
| 19 | + |
| 20 | +### `oci_genai_tracing.ipynb` |
| 21 | +Comprehensive Jupyter notebook demonstrating: |
| 22 | +- Basic non-streaming chat completions |
| 23 | +- Streaming chat completions |
| 24 | +- Advanced parameter configuration |
| 25 | +- Error handling |
| 26 | +- Multi-turn conversations |
| 27 | + |
| 28 | +### `simple_oci_example.py` |
| 29 | +Simple Python script for quick testing: |
| 30 | +```bash |
| 31 | +export OCI_COMPARTMENT_ID="ocid1.compartment.oc1..your-actual-ocid" |
| 32 | +python simple_oci_example.py |
| 33 | +``` |
| 34 | + |
| 35 | +## Quick Start |
| 36 | + |
| 37 | +### 1. Configure OCI |
| 38 | + |
| 39 | +Set up your OCI configuration using one of these methods: |
| 40 | + |
| 41 | +**Option A: OCI CLI Setup** |
| 42 | +```bash |
| 43 | +oci setup config |
| 44 | +``` |
| 45 | + |
| 46 | +**Option B: Environment Variables** |
| 47 | +```bash |
| 48 | +export OCI_CONFIG_FILE="~/.oci/config" |
| 49 | +export OCI_CONFIG_PROFILE="DEFAULT" |
| 50 | +``` |
| 51 | + |
| 52 | +**Option C: Instance Principal** (when running on OCI compute) |
| 53 | +```python |
| 54 | +from oci.auth.signers import InstancePrincipalsSecurityTokenSigner |
| 55 | +config = {} |
| 56 | +signer = InstancePrincipalsSecurityTokenSigner() |
| 57 | +``` |
| 58 | + |
| 59 | +### 2. Basic Usage |
| 60 | + |
| 61 | +```python |
| 62 | +import oci |
| 63 | +from oci.generative_ai_inference import GenerativeAiInferenceClient |
| 64 | +from oci.generative_ai_inference.models import ChatDetails, GenericChatRequest, Message |
| 65 | +from openlayer.lib.integrations import trace_oci_genai |
| 66 | + |
| 67 | +# Configure OCI client |
| 68 | +config = oci.config.from_file() |
| 69 | +client = GenerativeAiInferenceClient( |
| 70 | + config=config, |
| 71 | + service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com" |
| 72 | +) |
| 73 | + |
| 74 | +# Apply Openlayer tracing |
| 75 | +traced_client = trace_oci_genai(client) |
| 76 | + |
| 77 | +# Make a request |
| 78 | +chat_request = GenericChatRequest( |
| 79 | + messages=[Message(role="user", content="Hello, AI!")], |
| 80 | + model_id="cohere.command-r-plus", |
| 81 | + max_tokens=100, |
| 82 | + temperature=0.7 |
| 83 | +) |
| 84 | + |
| 85 | +chat_details = ChatDetails( |
| 86 | + compartment_id="your-compartment-ocid", |
| 87 | + chat_request=chat_request |
| 88 | +) |
| 89 | + |
| 90 | +response = traced_client.chat(chat_details, inference_id="my-custom-id") |
| 91 | +``` |
| 92 | + |
| 93 | +## Supported Models |
| 94 | + |
| 95 | +The integration supports all OCI Generative AI models including: |
| 96 | + |
| 97 | +### Cohere Models |
| 98 | +- `cohere.command-r-16k` - 16K context window |
| 99 | +- `cohere.command-r-plus` - Enhanced capabilities |
| 100 | + |
| 101 | +### Meta Llama Models |
| 102 | +- `meta.llama-3.1-70b-instruct` - 70B parameters, 128K context |
| 103 | +- `meta.llama-3.1-405b-instruct` - 405B parameters, largest available |
| 104 | + |
| 105 | +## Features Traced |
| 106 | + |
| 107 | +The Openlayer integration automatically captures: |
| 108 | + |
| 109 | +- ✅ **Request Details**: Model ID, parameters, messages |
| 110 | +- ✅ **Response Data**: Generated content, token usage |
| 111 | +- ✅ **Performance Metrics**: Latency, time to first token (streaming) |
| 112 | +- ✅ **Error Information**: When requests fail |
| 113 | +- ✅ **Custom Inference IDs**: For request tracking |
| 114 | +- ✅ **Model Parameters**: Temperature, top_p, max_tokens, etc. |
| 115 | + |
| 116 | +## Streaming Support |
| 117 | + |
| 118 | +Both streaming and non-streaming requests are fully supported: |
| 119 | + |
| 120 | +```python |
| 121 | +# Non-streaming |
| 122 | +chat_request = GenericChatRequest(..., is_stream=False) |
| 123 | +response = traced_client.chat(chat_details) |
| 124 | + |
| 125 | +# Streaming |
| 126 | +chat_request = GenericChatRequest(..., is_stream=True) |
| 127 | +for chunk in traced_client.chat(chat_details): |
| 128 | + print(chunk.data.choices[0].delta.content, end='') |
| 129 | +``` |
| 130 | + |
| 131 | +## Configuration Options |
| 132 | + |
| 133 | +### OCI Endpoints by Region |
| 134 | +- **US East (Ashburn)**: `https://inference.generativeai.us-ashburn-1.oci.oraclecloud.com` |
| 135 | +- **US West (Phoenix)**: `https://inference.generativeai.us-phoenix-1.oci.oraclecloud.com` |
| 136 | +- **UK South (London)**: `https://inference.generativeai.uk-london-1.oci.oraclecloud.com` |
| 137 | +- **Germany Central (Frankfurt)**: `https://inference.generativeai.eu-frankfurt-1.oci.oraclecloud.com` |
| 138 | + |
| 139 | +### Model Parameters |
| 140 | +```python |
| 141 | +GenericChatRequest( |
| 142 | + messages=[...], |
| 143 | + model_id="cohere.command-r-plus", |
| 144 | + max_tokens=500, # Maximum tokens to generate |
| 145 | + temperature=0.7, # Creativity (0.0-1.0) |
| 146 | + top_p=0.8, # Nucleus sampling |
| 147 | + top_k=40, # Top-k sampling |
| 148 | + frequency_penalty=0.2, # Reduce repetition |
| 149 | + presence_penalty=0.1, # Encourage new topics |
| 150 | + stop=["\n\n"], # Stop sequences |
| 151 | + is_stream=True # Enable streaming |
| 152 | +) |
| 153 | +``` |
| 154 | + |
| 155 | +## Error Handling |
| 156 | + |
| 157 | +The integration gracefully handles errors and traces them: |
| 158 | + |
| 159 | +```python |
| 160 | +try: |
| 161 | + response = traced_client.chat(chat_details) |
| 162 | +except oci.exceptions.ServiceError as e: |
| 163 | + print(f"OCI Service Error: {e}") |
| 164 | +except Exception as e: |
| 165 | + print(f"Unexpected error: {e}") |
| 166 | +# All errors are automatically traced by Openlayer |
| 167 | +``` |
| 168 | + |
| 169 | +## Best Practices |
| 170 | + |
| 171 | +1. **Use Custom Inference IDs**: For better tracking and debugging |
| 172 | +2. **Set Appropriate Timeouts**: For long-running requests |
| 173 | +3. **Monitor Token Usage**: To manage costs |
| 174 | +4. **Handle Rate Limits**: Implement retry logic |
| 175 | +5. **Secure Credentials**: Use IAM roles and policies |
| 176 | + |
| 177 | +## Troubleshooting |
| 178 | + |
| 179 | +### Common Issues |
| 180 | + |
| 181 | +**Config File Not Found** |
| 182 | +```bash |
| 183 | +oci setup config |
| 184 | +``` |
| 185 | + |
| 186 | +**Authentication Errors** |
| 187 | +```bash |
| 188 | +oci iam user get --user-id $(oci iam user list --query 'data[0].id' --raw-output) |
| 189 | +``` |
| 190 | + |
| 191 | +**Service Unavailable** |
| 192 | +- Check if Generative AI is available in your region |
| 193 | +- Verify compartment OCID is correct |
| 194 | +- Ensure proper IAM permissions |
| 195 | + |
| 196 | +**Import Errors** |
| 197 | +```bash |
| 198 | +pip install --upgrade oci openlayer |
| 199 | +``` |
| 200 | + |
| 201 | +## Support |
| 202 | + |
| 203 | +- **OCI Generative AI Documentation**: [docs.oracle.com](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm) |
| 204 | +- **Openlayer Documentation**: [openlayer.com/docs](https://openlayer.com/docs) |
| 205 | +- **OCI Python SDK**: [github.com/oracle/oci-python-sdk](https://github.com/oracle/oci-python-sdk) |
| 206 | + |
| 207 | +## License |
| 208 | + |
| 209 | +This integration follows the same license as the main Openlayer project. |
0 commit comments