Skip to content

Commit fead515

Browse files
howieleungCopilot
andauthored
Document tools (#43983)
* Document tools * update * Update sdk/ai/azure-ai-projects/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 8ad14f3 commit fead515

17 files changed

+458
-79
lines changed

sdk/ai/azure-ai-projects/README.md

Lines changed: 356 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,22 @@ The AI Projects client library (in preview) is part of the Microsoft Foundry SDK
44
resources in your Microsoft Foundry Project. Use it to:
55

66
* **Create and run Agents** using methods on methods on the `.agents` client property.
7+
* **Enhance Agents with specialized tools**:
8+
* Agent-to-Agent (A2A)
9+
* Azure AI Search
10+
* Bing Custom Search
11+
* Bing Grounding
12+
* Browser Automation
13+
* Code Interpreter
14+
* Computer Use
15+
* File Search
16+
* Function Tool
17+
* Image Generation
18+
* MCP with Project Connection
19+
* Microsoft Fabric
20+
* Model Context Protocol (MCP)
21+
* SharePoint
22+
* Web Search
723
* **Get an OpenAI client** using `.get_openai_client()` method to run "Responses" and "Conversations" operations with your Agent.
824
* **Manage memory stores** for Agent conversations, using the `.memory_store` operations.
925
* **Run Evaluations** to assess the performance of your generative AI application, using the `.evaluation_rules`,
@@ -178,9 +194,348 @@ print("Agent deleted")
178194

179195
<!-- END SNIPPET -->
180196

197+
### Using Agent tools
198+
199+
Agents can be enhanced with specialized tools for various capabilities. Tools are organized by their connection requirements:
200+
201+
#### Built-in Tools
202+
203+
These tools work immediately without requiring external connections.
204+
205+
* **Code Interpreter**
206+
207+
Write and run Python code in a sandboxed environment, process files and work with diverse data formats. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-code-interpreter)
208+
209+
<!-- SNIPPET:sample_agent_code_interpreter.tool_declaration -->
210+
211+
```python
212+
# Load the CSV file to be processed
213+
asset_file_path = os.path.abspath(
214+
os.path.join(os.path.dirname(__file__), "../assets/synthetic_500_quarterly_results.csv")
215+
)
216+
217+
# Upload the CSV file for the code interpreter
218+
file = openai_client.files.create(purpose="assistants", file=open(asset_file_path, "rb"))
219+
tool = CodeInterpreterTool(container=CodeInterpreterToolAuto(file_ids=[file.id]))
220+
```
221+
222+
<!-- END SNIPPET -->
223+
224+
*After calling `responses.create()`, check for generated files in response annotations (type `container_file_citation`) and download them using `openai_client.containers.files.content.retrieve()`.*
225+
226+
See the full sample code in [sample_agent_code_interpreter.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_code_interpreter.py).
227+
228+
* **File Search**
229+
230+
Built-in RAG (Retrieval-Augmented Generation) tool to process and search through documents using vector stores for knowledge retrieval. [OpenAI Documentation](https://platform.openai.com/docs/assistants/tools/file-search)
231+
232+
<!-- SNIPPET:sample_agent_file_search.tool_declaration -->
233+
234+
```python
235+
# Create vector store for file search
236+
vector_store = openai_client.vector_stores.create(name="ProductInfoStore")
237+
print(f"Vector store created (id: {vector_store.id})")
238+
239+
# Load the file to be indexed for search
240+
asset_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../assets/product_info.md"))
241+
242+
# Upload file to vector store
243+
file = openai_client.vector_stores.files.upload_and_poll(
244+
vector_store_id=vector_store.id, file=open(asset_file_path, "rb")
245+
)
246+
print(f"File uploaded to vector store (id: {file.id})")
247+
248+
tool = FileSearchTool(vector_store_ids=[vector_store.id])
249+
```
250+
251+
<!-- END SNIPPET -->
252+
253+
See the full sample code in [sample_agent_file_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_file_search.py).
254+
255+
* **Image Generation**
256+
257+
Generate images based on text prompts with customizable resolution, quality, and style settings:
258+
259+
<!-- SNIPPET:sample_agent_image_generation.tool_declaration -->
260+
261+
```python
262+
tool = ImageGenTool(quality="low", size="1024x1024")
263+
```
264+
265+
<!-- END SNIPPET -->
266+
267+
After calling `responses.create()`, you can download file using the returned response:
268+
<!-- SNIPPET:sample_agent_image_generation.download_image -->
269+
270+
```python
271+
image_data = [output.result for output in response.output if output.type == "image_generation_call"]
272+
273+
if image_data and image_data[0]:
274+
print("Downloading generated image...")
275+
filename = "microsoft.png"
276+
file_path = os.path.abspath(filename)
277+
278+
with open(file_path, "wb") as f:
279+
f.write(base64.b64decode(image_data[0]))
280+
```
281+
282+
<!-- END SNIPPET -->
283+
284+
See the full sample code in [sample_agent_image_generation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_image_generation.py).
285+
286+
287+
* **Web Search**
288+
289+
Perform general web searches to retrieve current information from the internet. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-web-search)
290+
291+
<!-- SNIPPET:sample_agent_web_search.tool_declaration -->
292+
293+
```python
294+
tool = WebSearchPreviewTool(user_location=ApproximateLocation(country="GB", city="London", region="London"))
295+
```
296+
297+
<!-- END SNIPPET -->
298+
299+
See the full sample code in [sample_agent_web_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_web_search.py).
300+
301+
* **Computer Use**
302+
303+
Enable agents to interact directly with computer systems for task automation and system operations:
304+
305+
<!-- SNIPPET:sample_agent_computer_use.tool_declaration -->
306+
307+
```python
308+
tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")
309+
```
310+
311+
<!-- END SNIPPET -->
312+
313+
*After calling `responses.create()`, process the response in an interaction loop. Handle `computer_call` output items and provide screenshots as `computer_call_output` with `computer_screenshot` type to continue the interaction.*
314+
315+
See the full sample code in [sample_agent_computer_use.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_computer_use.py).
316+
317+
* **Model Context Protocol (MCP)**
318+
319+
Integrate MCP servers to extend agent capabilities with standardized tools and resources. [OpenAI Documentation](https://platform.openai.com/docs/guides/tools-connectors-mcp)
320+
321+
<!-- SNIPPET:sample_agent_mcp.tool_declaration -->
322+
323+
```python
324+
mcp_tool = MCPTool(
325+
server_label="api-specs",
326+
server_url="https://gitmcp.io/Azure/azure-rest-api-specs",
327+
require_approval="always",
328+
)
329+
```
330+
331+
<!-- END SNIPPET -->
332+
333+
*After calling `responses.create()`, check for `mcp_approval_request` items in the response output. Send back `McpApprovalResponse` with your approval decision to allow the agent to continue its work.*
334+
335+
See the full sample code in [sample_agent_mcp.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_mcp.py).
336+
337+
338+
* **Function Tool**
339+
340+
Define custom functions that allow agents to interact with external APIs, databases, or application logic. [OpenAI Documentation](https://platform.openai.com/docs/guides/function-calling)
341+
342+
<!-- SNIPPET:sample_agent_function_tool.tool_declaration -->
343+
344+
```python
345+
tool = FunctionTool(
346+
name="get_horoscope",
347+
parameters={
348+
"type": "object",
349+
"properties": {
350+
"sign": {
351+
"type": "string",
352+
"description": "An astrological sign like Taurus or Aquarius",
353+
},
354+
},
355+
"required": ["sign"],
356+
"additionalProperties": False,
357+
},
358+
description="Get today's horoscope for an astrological sign.",
359+
strict=True,
360+
)
361+
```
362+
363+
<!-- END SNIPPET -->
364+
365+
*After calling `responses.create()`, process `function_call` items from response output, execute your function logic with the provided arguments, and send back `FunctionCallOutput` with the results.*
366+
367+
See the full sample code in [sample_agent_function_tool.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_function_tool.py).
368+
369+
#### Connection-Based Tools
370+
371+
These tools require configuring connections in your AI Foundry project and use `project_connection_id`.
372+
373+
* **Azure AI Search**
374+
375+
Integrate with Azure AI Search indexes for powerful knowledge retrieval and semantic search capabilities:
376+
377+
<!-- SNIPPET:sample_agent_ai_search.tool_declaration -->
378+
379+
```python
380+
tool = AzureAISearchAgentTool(
381+
azure_ai_search=AzureAISearchToolResource(
382+
indexes=[
383+
AISearchIndexResource(
384+
project_connection_id=os.environ["AI_SEARCH_PROJECT_CONNECTION_ID"],
385+
index_name=os.environ["AI_SEARCH_INDEX_NAME"],
386+
query_type=AzureAISearchQueryType.SIMPLE,
387+
),
388+
]
389+
)
390+
)
391+
```
392+
393+
<!-- END SNIPPET -->
394+
395+
See the full sample code in [sample_agent_ai_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_ai_search.py).
396+
397+
* **Bing Grounding**
398+
399+
Ground agent responses with real-time web search results from Bing to provide up-to-date information:
400+
401+
<!-- SNIPPET:sample_agent_bing_grounding.tool_declaration -->
402+
403+
```python
404+
tool = BingGroundingAgentTool(
405+
bing_grounding=BingGroundingSearchToolParameters(
406+
search_configurations=[
407+
BingGroundingSearchConfiguration(project_connection_id=os.environ["BING_PROJECT_CONNECTION_ID"])
408+
]
409+
)
410+
)
411+
```
412+
413+
<!-- END SNIPPET -->
414+
415+
See the full sample code in [sample_agent_bing_grounding.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_bing_grounding.py).
416+
417+
* **Bing Custom Search**
418+
419+
Use custom-configured Bing search instances for domain-specific or filtered web search results:
420+
421+
<!-- SNIPPET:sample_agent_bing_custom_search.tool_declaration -->
422+
423+
```python
424+
tool = BingCustomSearchAgentTool(
425+
bing_custom_search_preview=BingCustomSearchToolParameters(
426+
search_configurations=[
427+
BingCustomSearchConfiguration(
428+
project_connection_id=os.environ["BING_CUSTOM_SEARCH_PROJECT_CONNECTION_ID"],
429+
instance_name=os.environ["BING_CUSTOM_SEARCH_INSTANCE_NAME"],
430+
)
431+
]
432+
)
433+
)
434+
```
435+
436+
<!-- END SNIPPET -->
437+
438+
See the full sample code in [sample_agent_bing_custom_search.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_bing_custom_search.py).
439+
440+
* **Microsoft Fabric**
441+
442+
Connect to and query Microsoft Fabric:
443+
444+
<!-- SNIPPET:sample_agent_fabric.tool_declaration -->
445+
446+
```python
447+
tool = MicrosoftFabricAgentTool(
448+
fabric_dataagent_preview=FabricDataAgentToolParameters(
449+
project_connections=[ToolProjectConnection(project_connection_id=os.environ["FABRIC_PROJECT_CONNECTION_ID"])]
450+
)
451+
)
452+
```
453+
454+
<!-- END SNIPPET -->
455+
456+
See the full sample code in [sample_agent_fabric.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_fabric.py).
457+
458+
* **SharePoint**
459+
460+
Access and search SharePoint documents, lists, and sites for enterprise knowledge integration:
461+
462+
<!-- SNIPPET:sample_agent_sharepoint.tool_declaration -->
463+
464+
```python
465+
tool = SharepointAgentTool(
466+
sharepoint_grounding_preview=SharepointGroundingToolParameters(
467+
project_connections=[
468+
ToolProjectConnection(project_connection_id=os.environ["SHAREPOINT_PROJECT_CONNECTION_ID"])
469+
]
470+
)
471+
)
472+
```
473+
474+
<!-- END SNIPPET -->
475+
476+
See the full sample code in [sample_agent_sharepoint.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_sharepoint.py).
477+
478+
* **Browser Automation**
479+
480+
Automate browser interactions for web scraping, testing, and interaction with web applications:
481+
482+
<!-- SNIPPET:sample_agent_browser_automation.tool_declaration -->
483+
484+
```python
485+
tool = BrowserAutomationAgentTool(
486+
browser_automation_preview=BrowserAutomationToolParameters(
487+
connection=BrowserAutomationToolConnectionParameters(
488+
project_connection_id=os.environ["BROWSER_AUTOMATION_PROJECT_CONNECTION_ID"],
489+
)
490+
)
491+
)
492+
```
493+
494+
<!-- END SNIPPET -->
495+
496+
See the full sample code in [sample_agent_browser_automation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_browser_automation.py).
497+
498+
499+
* **MCP with Project Connection**
500+
501+
MCP integration using project-specific connections for accessing connected MCP servers:
502+
503+
<!-- SNIPPET:sample_agent_mcp_with_project_connection.tool_declaration -->
504+
505+
```python
506+
tool = MCPTool(
507+
server_label="api-specs",
508+
server_url="https://api.githubcopilot.com/mcp",
509+
require_approval="always",
510+
project_connection_id=os.environ["MCP_PROJECT_CONNECTION_ID"],
511+
)
512+
```
513+
514+
<!-- END SNIPPET -->
515+
516+
See the full sample code in [sample_agent_mcp_with_project_connection.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_mcp_with_project_connection.py).
517+
518+
* **Agent-to-Agent (A2A)**
519+
520+
Enable multi-agent collaboration where agents can communicate and delegate tasks to other specialized agents:
521+
522+
<!-- SNIPPET:sample_agent_to_agent.tool_declaration -->
523+
524+
```python
525+
tool = A2ATool(
526+
project_connection_id=os.environ["A2A_PROJECT_CONNECTION_ID"],
527+
)
528+
```
529+
530+
<!-- END SNIPPET -->
531+
532+
See the full sample code in [sample_agent_to_agent.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools/sample_agent_to_agent.py).
533+
534+
For complete working examples of all tools, see the [sample tools directory](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/tools).
535+
181536
### Evaluation
182537

183-
Evaluation in Azure AI Project client library provides quantitive, AI-assisted quality and safety metrics to asses performance and Evaluate LLM Models, GenAI Application and Agents. Metrics are defined as evaluators. Built-in or custom evaluators can provide comprehensive evaluation insights.
538+
Evaluation in Azure AI Project client library provides quantitative, AI-assisted quality and safety metrics to asses performance and Evaluate LLM Models, GenAI Application and Agents. Metrics are defined as evaluators. Built-in or custom evaluators can provide comprehensive evaluation insights.
184539

185540
The code below shows some evaluation operations. Full list of sample can be found under "evaluation" folder in the [package samples][samples]
186541

0 commit comments

Comments
 (0)