You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: sdk/ai/azure-ai-projects/README.md
+356-1Lines changed: 356 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,22 @@ The AI Projects client library (in preview) is part of the Microsoft Foundry SDK
4
4
resources in your Microsoft Foundry Project. Use it to:
5
5
6
6
***Create and run Agents** using methods on methods on the `.agents` client property.
7
+
***Enhance Agents with specialized tools**:
8
+
* Agent-to-Agent (A2A)
9
+
* Azure AI Search
10
+
* Bing Custom Search
11
+
* Bing Grounding
12
+
* Browser Automation
13
+
* Code Interpreter
14
+
* Computer Use
15
+
* File Search
16
+
* Function Tool
17
+
* Image Generation
18
+
* MCP with Project Connection
19
+
* Microsoft Fabric
20
+
* Model Context Protocol (MCP)
21
+
* SharePoint
22
+
* Web Search
7
23
***Get an OpenAI client** using `.get_openai_client()` method to run "Responses" and "Conversations" operations with your Agent.
8
24
***Manage memory stores** for Agent conversations, using the `.memory_store` operations.
9
25
***Run Evaluations** to assess the performance of your generative AI application, using the `.evaluation_rules`,
@@ -178,9 +194,348 @@ print("Agent deleted")
178
194
179
195
<!-- END SNIPPET -->
180
196
197
+
### Using Agent tools
198
+
199
+
Agents can be enhanced with specialized tools for various capabilities. Tools are organized by their connection requirements:
200
+
201
+
#### Built-in Tools
202
+
203
+
These tools work immediately without requiring external connections.
204
+
205
+
***Code Interpreter**
206
+
207
+
Write and run Python code in a sandboxed environment, process files and work with diverse data formats. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-code-interpreter)
*After calling `responses.create()`, check for generated files in response annotations (type `container_file_citation`) and download them using `openai_client.containers.files.content.retrieve()`.*
225
+
226
+
See the full sample code in [sample_agent_code_interpreter.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_code_interpreter.py).
227
+
228
+
***File Search**
229
+
230
+
Built-in RAG (Retrieval-Augmented Generation) tool to process and search through documents using vector stores for knowledge retrieval. [OpenAI Documentation](https://platform.openai.com/docs/assistants/tools/file-search)
See the full sample code in [sample_agent_file_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_file_search.py).
254
+
255
+
***Image Generation**
256
+
257
+
Generate images based on text prompts with customizable resolution, quality, and style settings:
image_data = [output.result for output in response.output if output.type =="image_generation_call"]
272
+
273
+
if image_data and image_data[0]:
274
+
print("Downloading generated image...")
275
+
filename ="microsoft.png"
276
+
file_path = os.path.abspath(filename)
277
+
278
+
withopen(file_path, "wb") as f:
279
+
f.write(base64.b64decode(image_data[0]))
280
+
```
281
+
282
+
<!-- END SNIPPET -->
283
+
284
+
See the full sample code in [sample_agent_image_generation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_image_generation.py).
285
+
286
+
287
+
***Web Search**
288
+
289
+
Perform general web searches to retrieve current information from the internet. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-web-search)
See the full sample code in [sample_agent_web_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_web_search.py).
300
+
301
+
***Computer Use**
302
+
303
+
Enable agents to interact directly with computer systems for task automation and system operations:
*After calling `responses.create()`, process the response in an interaction loop. Handle `computer_call` output items and provide screenshots as `computer_call_output` with `computer_screenshot` type to continue the interaction.*
314
+
315
+
See the full sample code in [sample_agent_computer_use.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_computer_use.py).
316
+
317
+
***Model Context Protocol (MCP)**
318
+
319
+
Integrate MCP servers to extend agent capabilities with standardized tools and resources. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-connectors-mcp)
*After calling `responses.create()`, check for `mcp_approval_request` items in the response output. Send back `McpApprovalResponse` with your approval decision to allow the agent to continue its work.*
334
+
335
+
See the full sample code in [sample_agent_mcp.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_mcp.py).
336
+
337
+
338
+
***Function Tool**
339
+
340
+
Define custom functions that allow agents to interact with external APIs, databases, or application logic. [OpenAI Documentation](https://platform.openai.com/docs/guides/function-calling)
"description": "An astrological sign like Taurus or Aquarius",
353
+
},
354
+
},
355
+
"required": ["sign"],
356
+
"additionalProperties": False,
357
+
},
358
+
description="Get today's horoscope for an astrological sign.",
359
+
strict=True,
360
+
)
361
+
```
362
+
363
+
<!-- END SNIPPET -->
364
+
365
+
*After calling `responses.create()`, process `function_call` items from response output, execute your function logic with the provided arguments, and send back `FunctionCallOutput` with the results.*
366
+
367
+
See the full sample code in [sample_agent_function_tool.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_function_tool.py).
368
+
369
+
#### Connection-Based Tools
370
+
371
+
These tools require configuring connections in your AI Foundry project and use `project_connection_id`.
372
+
373
+
***Azure AI Search**
374
+
375
+
Integrate with Azure AI Search indexes for powerful knowledge retrieval and semantic search capabilities:
See the full sample code in [sample_agent_ai_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_ai_search.py).
396
+
397
+
***Bing Grounding**
398
+
399
+
Ground agent responses with real-time web search results from Bing to provide up-to-date information:
See the full sample code in [sample_agent_bing_grounding.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_bing_grounding.py).
416
+
417
+
***Bing Custom Search**
418
+
419
+
Use custom-configured Bing search instances for domain-specific or filtered web search results:
See the full sample code in [sample_agent_bing_custom_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_bing_custom_search.py).
See the full sample code in [sample_agent_fabric.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_fabric.py).
457
+
458
+
***SharePoint**
459
+
460
+
Access and search SharePoint documents, lists, and sites for enterprise knowledge integration:
See the full sample code in [sample_agent_sharepoint.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_sharepoint.py).
477
+
478
+
***Browser Automation**
479
+
480
+
Automate browser interactions for web scraping, testing, and interaction with web applications:
See the full sample code in [sample_agent_browser_automation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_browser_automation.py).
497
+
498
+
499
+
***MCP with Project Connection**
500
+
501
+
MCP integration using project-specific connections for accessing connected MCP servers:
See the full sample code in [sample_agent_mcp_with_project_connection.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_mcp_with_project_connection.py).
517
+
518
+
***Agent-to-Agent (A2A)**
519
+
520
+
Enable multi-agent collaboration where agents can communicate and delegate tasks to other specialized agents:
See the full sample code in [sample_agent_to_agent.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_to_agent.py).
533
+
534
+
For complete working examples of all tools, see the [sample tools directory](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools).
535
+
181
536
### Evaluation
182
537
183
-
Evaluation in Azure AI Project client library provides quantitive, AI-assisted quality and safety metrics to asses performance and Evaluate LLM Models, GenAI Application and Agents. Metrics are defined as evaluators. Built-in or custom evaluators can provide comprehensive evaluation insights.
538
+
Evaluation in Azure AI Project client library provides quantitative, AI-assisted quality and safety metrics to asses performance and Evaluate LLM Models, GenAI Application and Agents. Metrics are defined as evaluators. Built-in or custom evaluators can provide comprehensive evaluation insights.
184
539
185
540
The code below shows some evaluation operations. Full list of sample can be found under "evaluation" folder in the [package samples][samples]
0 commit comments