From 3b2392821c593a3064581b7947c8c248b90ea0fb Mon Sep 17 00:00:00 2001 From: Manni Arora Date: Mon, 4 Aug 2025 15:50:32 -0700 Subject: [PATCH 1/3] Updated Transparency doc --- Agent_Based_Samples/customer_assist/README.md | 3 + .../customer_assist/TRANSPARENCY-FAQ.md | 75 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md diff --git a/Agent_Based_Samples/customer_assist/README.md b/Agent_Based_Samples/customer_assist/README.md index bb15861..3ccf542 100644 --- a/Agent_Based_Samples/customer_assist/README.md +++ b/Agent_Based_Samples/customer_assist/README.md @@ -88,6 +88,9 @@ The datasets ([dataset-1](src/solution_accelerators/customer_assist/data) and [d This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. +# Responsible AI Transparency FAQ +Please refer to [Transparency FAQ](TRANSPARENCY-FAQ.md) for responsible AI transparency details of this solution accelerator. + ---
diff --git a/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md b/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md new file mode 100644 index 0000000..4d81d2e --- /dev/null +++ b/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md @@ -0,0 +1,75 @@ +# Customer Assist Solution Accelerator: Responsible AI FAQ + +## What is Customer Support Solution Accelerator? + +The Customer Support Solution Accelerator (also called "Customer Assist") is an enterprise-grade AI-powered solution that demonstrates how intelligent agent orchestration can transform customer service operations. It leverages the Microsoft Semantic Kernel Process Framework and Azure AI Foundry to empower customer service representatives with real-time insights, contextual assistance, and automated workflows. + +## What can the Customer Support Solution Accelerator do? + +The solution provides several key capabilities: + +### Core Features: + +- **Multi-Agent Orchestration**: Codifies business processes using specialized agents (Conversation Processor, Sentiment Analysis, and Post-call Analysis agents) orchestrated with Microsoft Semantic Kernel +- **Multi-Modal Support**: Handles text, images, audio, and documents in real-time with content understanding and structured insight extraction +- **Flexible AI Integration**: Supports any language model ("Bring Your Own LLM") with task allocation to specialized models for cost and performance optimization +- **Real-time Processing**: Provides live sentiment analysis, document verification, and policy retrieval +- **Agent Evaluation & Observability**: Monitors agent behavior and system health with usage, performance, and quality analytics + +### Business Capabilities: + +- **Enhanced Customer Experience**: Real-time sentiment analysis and personalized interactions +- **Improved Operational Efficiency**: Automated document verification and policy retrieval +- **Reduced Training Requirements**: Enables new representatives to operate with expert-level knowledge from day one +- **Increased First-Contact Resolution**: Addresses customer needs faster and more effectively + +## What is Customer Support Solution Accelerator intended use(s)? + +The solution is designed for multiple industry verticals: + +- **Banking & Financial Services**: Streamline loan processing, fraud resolution, and financial product guidance +- **Healthcare & Insurance**: Automate onboarding, claims assistance, and ensure compliance during patient/member interactions +- **Retail & eCommerce**: Enhance product discovery, resolve order issues, and drive upsell opportunities +- **Telecom & Utilities**: Guide agents through technical troubleshooting, plan recommendations, and outage communication +- **Travel & Hospitality**: Support booking changes, handle complaints sensitively, and personalize loyalty program engagement + +## How was Customer Support Accelerator evaluated? What metrics are used to measure performance? + +The solution incorporates a comprehensive evaluation framework powered by the Azure AI Evaluation SDK, enabling performance evaluations across agents. + +### Evaluation Metrics: + +- **Intent Resolution**: Measures the agent's ability to accurately identify and scope user intent. Scale: 1–5 (higher is better) +- **Coherence**: Measures how well the language model can produce output that flows smoothly, reads naturally, and resembles human-like language. Scale: 1–5 (higher is better) + +We've applied this metric to evaluate the Assist Agent, verifying whether the agent correctly interprets and acts on user intent, and produces relevant & coherent responses. + +For safety evaluations, we've used the Azure AI Foundry Red Teaming Agent, which simulates a variety of attack strategies and complexity levels to test the system end-to-end. + +### Key safeguards in place: + +- **Text and Image Moderators**: All user inputs are first screened by dedicated moderation layers leveraging Azure AI Content Safety Service. +- **Azure OpenAI Content Filters**: Every agent uses Azure OpenAI service with content filters enabled at the model deployment level. + +## What are the limitations of Customer Support Solution Accelerator? + +### Performance Dependencies: + +- Azure service throttling +- Network latency requirements + +## Sensitive Use Guidance + +We encourage users to incorporate the below considerations when leveraging the workflow for the scenarios below. + +**Banking & Financial Services**: AI-generated financial guidance should include disclaimers and source attribution to the extent possible. Users should be able to trace advice to validated financial documents or policies. Confidence scores for fraud detection or loan eligibility should be interpretable, with thresholds triggering mandatory human review when confidence is low. Agent suggestions should be explainable, with links to relevant regulations or internal policies. + +**Healthcare & Insurance**: Claims assistance and onboarding should clearly indicate machine assistance, with all outputs reviewed by authorized personnel. Sensitive health information must be processed under strict data minimization principles, with access controls and retention policies enforced. AI-generated summaries should preserve the original message and tone, especially when dealing with urgent or emotionally charged content. + +**Retail & eCommerce**: Product recommendations and upsell prompts should be explainable and grounded in catalog data. AI-generated responses to order issues must be labeled and editable by human customer assistant agents. Observability metrics should track ungroundness rates and user satisfaction to ensure transparency in customer interactions. + +**Telecom & Utilities**: Troubleshooting flows should include transparency on decision logic. Plan recommendations should link to terms and conditions and be accompanied by a "Why was this suggested?" feature. All AI-generated actions should be logged and auditable to support regulatory compliance. + +**Travel & Hospitality**: Complaint handling and loyalty engagement should be sensitive to tone and context. AI-generated responses should be labeled and editable, with clear indicators of machine assistance. Booking changes and itinerary suggestions should be traceable to source data and reviewed by human agents before finalization. + +For key use cases that highlight customer-impacting actions, we emphasize making outputs available to human stakeholders so that they can be reviewed regularly. \ No newline at end of file From 2bb4603b6ed9081264ee266ac360b88c8c725e08 Mon Sep 17 00:00:00 2001 From: Manni Arora Date: Mon, 4 Aug 2025 15:53:23 -0700 Subject: [PATCH 2/3] Clean up --- Agent_Based_Samples/release_manager/scripts/createEnv.ps1 | 7 ------- Agent_Based_Samples/sales_analyst/scripts/createEnv.ps1 | 7 ------- 2 files changed, 14 deletions(-) diff --git a/Agent_Based_Samples/release_manager/scripts/createEnv.ps1 b/Agent_Based_Samples/release_manager/scripts/createEnv.ps1 index 0196ec1..746fdec 100644 --- a/Agent_Based_Samples/release_manager/scripts/createEnv.ps1 +++ b/Agent_Based_Samples/release_manager/scripts/createEnv.ps1 @@ -32,13 +32,6 @@ if (-Not (Test-Path -Path "$Path/.venv/pip.ini")) { # now create the pip.ini file Write-Host "Creating pip.ini file in directory $Path/.venv and set up the index-url" - $pipIniContent = @" -[global] -index-url=https://pkgs.dev.azure.com/AIP-CXE-Eng/Engineering/_packaging/CustomerEng/pypi/simple/ -"@ - - $pipIniPath = Join-Path -Path "$Path/.venv" -ChildPath "pip.ini" - $pipIniContent | Out-File -FilePath $pipIniPath -Encoding UTF8 Write-Host "Python virtual environment setup completed successfully." diff --git a/Agent_Based_Samples/sales_analyst/scripts/createEnv.ps1 b/Agent_Based_Samples/sales_analyst/scripts/createEnv.ps1 index 0196ec1..746fdec 100644 --- a/Agent_Based_Samples/sales_analyst/scripts/createEnv.ps1 +++ b/Agent_Based_Samples/sales_analyst/scripts/createEnv.ps1 @@ -32,13 +32,6 @@ if (-Not (Test-Path -Path "$Path/.venv/pip.ini")) { # now create the pip.ini file Write-Host "Creating pip.ini file in directory $Path/.venv and set up the index-url" - $pipIniContent = @" -[global] -index-url=https://pkgs.dev.azure.com/AIP-CXE-Eng/Engineering/_packaging/CustomerEng/pypi/simple/ -"@ - - $pipIniPath = Join-Path -Path "$Path/.venv" -ChildPath "pip.ini" - $pipIniContent | Out-File -FilePath $pipIniPath -Encoding UTF8 Write-Host "Python virtual environment setup completed successfully." From e11bf373b1cb7c965c333dc629ab923d05a09f36 Mon Sep 17 00:00:00 2001 From: Manni Arora Date: Mon, 4 Aug 2025 15:55:46 -0700 Subject: [PATCH 3/3] Clean up --- .../customer_assist/TRANSPARENCY-FAQ.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md b/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md index 4d81d2e..404adc0 100644 --- a/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md +++ b/Agent_Based_Samples/customer_assist/TRANSPARENCY-FAQ.md @@ -1,10 +1,10 @@ # Customer Assist Solution Accelerator: Responsible AI FAQ -## What is Customer Support Solution Accelerator? +## What is Customer Assist Solution Accelerator? -The Customer Support Solution Accelerator (also called "Customer Assist") is an enterprise-grade AI-powered solution that demonstrates how intelligent agent orchestration can transform customer service operations. It leverages the Microsoft Semantic Kernel Process Framework and Azure AI Foundry to empower customer service representatives with real-time insights, contextual assistance, and automated workflows. +The Customer Assist Solution Accelerator (also called "Customer Assist") is an enterprise-grade AI-powered solution that demonstrates how intelligent agent orchestration can transform customer service operations. It leverages the Microsoft Semantic Kernel Process Framework and Azure AI Foundry to empower customer service representatives with real-time insights, contextual assistance, and automated workflows. -## What can the Customer Support Solution Accelerator do? +## What can the Customer Assist Solution Accelerator do? The solution provides several key capabilities: @@ -23,7 +23,7 @@ The solution provides several key capabilities: - **Reduced Training Requirements**: Enables new representatives to operate with expert-level knowledge from day one - **Increased First-Contact Resolution**: Addresses customer needs faster and more effectively -## What is Customer Support Solution Accelerator intended use(s)? +## What is Customer Assist Solution Accelerator intended use(s)? The solution is designed for multiple industry verticals: @@ -33,7 +33,7 @@ The solution is designed for multiple industry verticals: - **Telecom & Utilities**: Guide agents through technical troubleshooting, plan recommendations, and outage communication - **Travel & Hospitality**: Support booking changes, handle complaints sensitively, and personalize loyalty program engagement -## How was Customer Support Accelerator evaluated? What metrics are used to measure performance? +## How was Customer Assist Accelerator evaluated? What metrics are used to measure performance? The solution incorporates a comprehensive evaluation framework powered by the Azure AI Evaluation SDK, enabling performance evaluations across agents. @@ -51,7 +51,7 @@ For safety evaluations, we've used the Azure AI Foundry Red Teaming Agent, which - **Text and Image Moderators**: All user inputs are first screened by dedicated moderation layers leveraging Azure AI Content Safety Service. - **Azure OpenAI Content Filters**: Every agent uses Azure OpenAI service with content filters enabled at the model deployment level. -## What are the limitations of Customer Support Solution Accelerator? +## What are the limitations of Customer Assist Solution Accelerator? ### Performance Dependencies: