|
| 1 | +# OpenAI Request Input Field Structure (On-the-Wire Format) |
| 2 | + |
| 3 | +This document describes the actual JSON structure sent to the OpenAI API in the `input` field of [`OpenAIRequest`](../pkg/aiusechat/openai/openai-convertmessage.go:111). |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The `input` field is a JSON array containing one of three object types: |
| 8 | + |
| 9 | +1. **Messages** (user/assistant) - `OpenAIMessage` objects |
| 10 | +2. **Function Calls** (tool invocations) - `OpenAIFunctionCallInput` objects |
| 11 | +3. **Function Call Results** (tool outputs) - `OpenAIFunctionCallOutputInput` objects |
| 12 | + |
| 13 | +These are converted from [`OpenAIChatMessage`](../pkg/aiusechat/openai/openai-backend.go:46-52) internal format and cleaned before transmission ([see lines 485-494](../pkg/aiusechat/openai/openai-backend.go:485-494)). |
| 14 | + |
| 15 | +## 1. Message Objects (User/Assistant) |
| 16 | + |
| 17 | +User and assistant messages sent as [`OpenAIMessage`](../pkg/aiusechat/openai/openai-backend.go:54-57): |
| 18 | + |
| 19 | +```json |
| 20 | +{ |
| 21 | + "role": "user", |
| 22 | + "content": [ |
| 23 | + { |
| 24 | + "type": "input_text", |
| 25 | + "text": "Hello, analyze this image" |
| 26 | + }, |
| 27 | + { |
| 28 | + "type": "input_image", |
| 29 | + "image_url": "data:image/png;base64,iVBORw0KG..." |
| 30 | + } |
| 31 | + ] |
| 32 | +} |
| 33 | +``` |
| 34 | + |
| 35 | +**Key Points:** |
| 36 | +- `role`: Always `"user"` or `"assistant"` |
| 37 | +- `content`: **Always an array** of content blocks (never a plain string) |
| 38 | + |
| 39 | +### Content Block Types |
| 40 | + |
| 41 | +#### Text Block |
| 42 | +```json |
| 43 | +{ |
| 44 | + "type": "input_text", |
| 45 | + "text": "message content here" |
| 46 | +} |
| 47 | +``` |
| 48 | + |
| 49 | +#### Image Block |
| 50 | +```json |
| 51 | +{ |
| 52 | + "type": "input_image", |
| 53 | + "image_url": "data:image/png;base64,..." |
| 54 | +} |
| 55 | +``` |
| 56 | +- Can be a data URL or https:// URL |
| 57 | +- `filename` field is **removed** during cleaning |
| 58 | + |
| 59 | +#### PDF File Block |
| 60 | +```json |
| 61 | +{ |
| 62 | + "type": "input_file", |
| 63 | + "file_data": "JVBERi0xLjQKJeLjz9M...", |
| 64 | + "filename": "document.pdf" |
| 65 | +} |
| 66 | +``` |
| 67 | +- `file_data`: Base64-encoded PDF content |
| 68 | + |
| 69 | +#### Function Call Block (in assistant messages) |
| 70 | +```json |
| 71 | +{ |
| 72 | + "type": "function_call", |
| 73 | + "call_id": "call_abc123", |
| 74 | + "name": "search_files", |
| 75 | + "arguments": {"query": "test"} |
| 76 | +} |
| 77 | +``` |
| 78 | + |
| 79 | +## 2. Function Call Objects (Tool Invocations) |
| 80 | + |
| 81 | +Tool calls from the model sent as [`OpenAIFunctionCallInput`](../pkg/aiusechat/openai/openai-backend.go:59-67): |
| 82 | + |
| 83 | +```json |
| 84 | +{ |
| 85 | + "type": "function_call", |
| 86 | + "call_id": "call_abc123", |
| 87 | + "name": "search_files", |
| 88 | + "arguments": "{\"query\":\"test\",\"path\":\"./src\"}" |
| 89 | +} |
| 90 | +``` |
| 91 | + |
| 92 | +**Key Points:** |
| 93 | +- `type`: Always `"function_call"` |
| 94 | +- `call_id`: Unique identifier generated by model |
| 95 | +- `name`: Function name to execute |
| 96 | +- `arguments`: JSON-encoded string of parameters |
| 97 | +- `status`: Optional (`"in_progress"`, `"completed"`, `"incomplete"`) |
| 98 | +- Internal `toolusedata` field is **removed** during cleaning |
| 99 | + |
| 100 | +## 3. Function Call Output Objects (Tool Results) |
| 101 | + |
| 102 | +Tool execution results sent as [`OpenAIFunctionCallOutputInput`](../pkg/aiusechat/openai/openai-backend.go:69-75): |
| 103 | + |
| 104 | +```json |
| 105 | +{ |
| 106 | + "type": "function_call_output", |
| 107 | + "call_id": "call_abc123", |
| 108 | + "output": "Found 3 files matching query" |
| 109 | +} |
| 110 | +``` |
| 111 | + |
| 112 | +**Key Points:** |
| 113 | +- `type`: Always `"function_call_output"` |
| 114 | +- `call_id`: Must match the original function call's `call_id` |
| 115 | +- `output`: Can be text, image array, or error object |
| 116 | + |
| 117 | +### Output Value Types |
| 118 | + |
| 119 | +#### Text Output |
| 120 | +```json |
| 121 | +{ |
| 122 | + "type": "function_call_output", |
| 123 | + "call_id": "call_abc123", |
| 124 | + "output": "Result text here" |
| 125 | +} |
| 126 | +``` |
| 127 | + |
| 128 | +#### Image Output |
| 129 | +```json |
| 130 | +{ |
| 131 | + "type": "function_call_output", |
| 132 | + "call_id": "call_abc123", |
| 133 | + "output": [ |
| 134 | + { |
| 135 | + "type": "input_image", |
| 136 | + "image_url": "data:image/png;base64,..." |
| 137 | + } |
| 138 | + ] |
| 139 | +} |
| 140 | +``` |
| 141 | + |
| 142 | +#### Error Output |
| 143 | +```json |
| 144 | +{ |
| 145 | + "type": "function_call_output", |
| 146 | + "call_id": "call_abc123", |
| 147 | + "output": "{\"ok\":\"false\",\"error\":\"File not found\"}" |
| 148 | +} |
| 149 | +``` |
| 150 | +- Error output is a JSON-encoded string containing `ok` and `error` fields |
| 151 | + |
| 152 | +## Complete Example |
| 153 | + |
| 154 | +```json |
| 155 | +{ |
| 156 | + "model": "gpt-4o", |
| 157 | + "input": [ |
| 158 | + { |
| 159 | + "role": "user", |
| 160 | + "content": [ |
| 161 | + { |
| 162 | + "type": "input_text", |
| 163 | + "text": "What files are in src/?" |
| 164 | + } |
| 165 | + ] |
| 166 | + }, |
| 167 | + { |
| 168 | + "type": "function_call", |
| 169 | + "call_id": "call_xyz789", |
| 170 | + "name": "list_files", |
| 171 | + "arguments": "{\"path\":\"src/\"}" |
| 172 | + }, |
| 173 | + { |
| 174 | + "type": "function_call_output", |
| 175 | + "call_id": "call_xyz789", |
| 176 | + "output": "main.go\nutil.go\nconfig.go" |
| 177 | + }, |
| 178 | + { |
| 179 | + "role": "assistant", |
| 180 | + "content": [ |
| 181 | + { |
| 182 | + "type": "output_text", |
| 183 | + "text": "The src/ directory contains 3 files: main.go, util.go, and config.go" |
| 184 | + } |
| 185 | + ] |
| 186 | + } |
| 187 | + ], |
| 188 | + "stream": true, |
| 189 | + "max_output_tokens": 4096 |
| 190 | +} |
| 191 | +``` |
| 192 | + |
| 193 | +## Cleaning Process |
| 194 | + |
| 195 | +Before transmission, internal fields are removed ([cleanup code](../pkg/aiusechat/openai/openai-backend.go:485-494)): |
| 196 | + |
| 197 | +- **Messages**: `previewurl` field removed, `filename` removed from `input_image` blocks |
| 198 | +- **Function Calls**: `toolusedata` field removed |
| 199 | +- **Function Outputs**: Sent as-is (no cleaning needed) |
| 200 | + |
| 201 | +This ensures the API receives only the fields it expects. |
0 commit comments