Merge branch 'develop'

Author: Bob Strahan

.gitlab-ci.yml

@@ -16,6 +16,7 @@ image: public.ecr.aws/docker/library/python:3.13-bookworm
 
 stages:
   - developer_tests
+  - deployment_validation
   - integration_tests
 
 developer_tests:
@@ -93,4 +94,23 @@ integration_tests:
     - poetry install
     - make put
     - make wait
+
+deployment_validation:
+  stage: deployment_validation
+  rules:
+    - when: always
+
+  before_script:
+    - apt-get update -y
+    - apt-get install curl unzip python3-pip -y
+    # Install AWS CLI
+    - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+    - unzip awscliv2.zip
+    - ./aws/install
+    # Install PyYAML for template analysis
+    - pip install PyYAML
+
+  script:
+    # Check if service role has sufficient permissions for main stack deployment
+    - python3 scripts/validate_service_role_permissions.py
 

CHANGELOG.md

@@ -5,8 +5,28 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.3.17]
+
 ### Added
 
+- **Edit Sections Feature for Modifying Class/Type and Reprocessing Extraction**
+  - Added Edit Sections interface for Pattern-2 and Pattern-3 workflows with reprocessing optimization
+  - **Key Features**: Section management (create, update, delete), classification updates, page reassignment with overlap detection, real-time validation
+  - **Selective Reprocessing**: Only modified sections are reprocessed while preserving existing data for unmodified sections
+  - **Processing Pipeline**: All functions (OCR/Classification/Extraction/Assessment) automatically skip redundant operations based on data presence
+  - **Pattern Compatibility**: Full functionality for Pattern-2/Pattern-3, informative modal for Pattern-1 explaining BDA not yet supported
+
+- **Analytics Agent Schema Optimization for Improved Performance**
+  - **Embedded Database Overview**: Complete table listing and guidance embedded directly in system prompt (no tool call needed)
+  - **On-Demand Detailed Schemas**: `get_table_info(['specific_tables'])` loads detailed column information only for tables actually needed by the query
+  - **Significant Performance Gains**: Eliminates redundant tool calls on every query while maintaining token efficiency
+  - **Enhanced SQL Guidance**: Comprehensive Athena/Trino function reference with explicit PostgreSQL operator warnings to prevent common query failures like `~` regex operator mistakes
+  - **Faster Time-to-Query**: Agent has immediate access to table overview and can proceed directly to detailed schema loading for relevant tables
+
+### Fixed
+- Fix missing data in Glue tables when using a document class that contains a dash (-). 
+
+
 ## [0.3.16]
 
 ### Added

VERSION

@@ -1 +1 @@
-0.3.16
+0.3.17

docs/deployment.md

@@ -36,6 +36,7 @@ You need to have the following packages installed on your computer:
 4. python 3.11 or later
 5. A local Docker daemon
 6. Python packages for publish.py: `pip install boto3 rich typer PyYAML botocore setuptools`
+7. **Node.js 18+** and **npm** (required for UI validation in publish script)
 
 For guidance on setting up a development environment, see:
 - [Development Environment Setup Guide on Linux](./setup-development-env-linux.md) 
@@ -136,12 +137,12 @@ aws cloudformation update-stack \
 
 
 **Pattern Parameter Options:**
-* `Pattern1` - Packet or Media processing with Bedrock Data Automation (BDA)
+* `Pattern1 - Packet or Media processing with Bedrock Data Automation (BDA)`
   * Can use an existing BDA project or create a new demo project
-* `Pattern2` - Packet processing with Textract and Bedrock
+* `Pattern2 - Packet processing with Textract and Bedrock`
   * Supports both page-level and holistic classification
   * Recommended for first-time users
-* `Pattern3` - Packet processing with Textract, SageMaker(UDOP), and Bedrock
+* `Pattern3 - Packet processing with Textract, SageMaker(UDOP), and Bedrock`
   * Requires a UDOP model in S3 that will be deployed on SageMaker
 
 After deployment, check the Outputs tab in the CloudFormation console to find links to dashboards, buckets, workflows, and other solution resources.

docs/pattern-3.md

@@ -30,7 +30,7 @@ This pattern implements an intelligent document processing workflow that uses UD
 
 ## Fine tuning a UDOP model for classification
 
-See [Fine-Tuning Models on SageMaker](./fine-tune-sm-udop-classification/README.md) 
+See [Fine-Tuning Models on SageMaker](../patterns/pattern-3/fine-tune-sm-udop-classification/README.md) 
 
 Once you have trained the model, deploy the GenAIIDP stack for Pattern-3 using the path for your new fine tuned model.
 

docs/web-ui.md

@@ -26,6 +26,66 @@ The solution includes a responsive web-based user interface built with React tha
 - **Document Process Flow visualization** for detailed workflow execution monitoring and troubleshooting
 - **Document Analytics** for querying and visualizing processed document data
 
+## Edit Sections
+
+The Edit Sections feature provides an intelligent interface for modifying document section classifications and page assignments, with automatic reprocessing optimization for Pattern-2 and Pattern-3 workflows.
+
+### Key Capabilities
+
+- **Section Management**: Create, update, and delete document sections with validation
+- **Classification Updates**: Change section document types with real-time validation
+- **Page Reassignment**: Move pages between sections with overlap detection
+- **Intelligent Reprocessing**: Only modified sections are reprocessed, preserving existing data
+- **Immediate Feedback**: Status updates appear instantly in the UI
+- **Pattern Compatibility**: Available for Pattern-2 and Pattern-3, with informative guidance for Pattern-1
+
+### How to Use
+
+1. Navigate to a completed document's detail page
+2. In the "Document Sections" panel, click the "Edit Sections" button
+3. **For Pattern-2/Pattern-3**: Enter edit mode with inline editing capabilities
+4. **For Pattern-1**: View informative modal explaining BDA architecture differences
+
+#### Editing Workflow (Pattern-2/Pattern-3)
+
+1. **Edit Section Classifications**: Use dropdowns to change document types
+2. **Modify Page Assignments**: Edit comma-separated page IDs (e.g., "1, 2, 3")
+3. **Add New Sections**: Click "Add Section" for new document boundaries  
+4. **Delete Sections**: Use remove buttons to delete unnecessary sections
+5. **Validation**: Real-time validation prevents overlapping pages and invalid configurations
+6. **Submit Changes**: Click "Save & Process Changes" to trigger selective reprocessing
+
+### Processing Optimization
+
+The Edit Sections feature uses **2-phase schema knowledge optimization**:
+
+#### Phase 1: Frontend 
+- **Selective Payload**: Only sends sections that actually changed
+- **Validation Engine**: Prevents invalid configurations before submission
+
+#### Phase 2: Backend   
+- **Pipeline**: Processing functions automatically skip redundant operations
+  - **OCR**: Skips if pages already have OCR data
+  - **Classification**: Skips if pages already classified
+  - **Extraction**: Skips if sections have extraction data
+  - **Assessment**: Skips if extraction results contain assessment data
+- **Selective Reprocessing**: Only modified sections lose their data and get reprocessed
+
+### Pattern Compatibility
+
+#### Pattern-2 and Pattern-3 Support
+- **Full Functionality**: Complete edit capabilities with intelligent reprocessing
+- **Performance Optimization**: Automatic selective processing for efficiency  
+- **Data Preservation**: Unmodified sections retain all processing results
+
+#### Pattern-1 Information
+Pattern-1 uses **Bedrock Data Automation (BDA)** with automatic section management. When Edit Sections is clicked, users see an informative modal explaining:
+
+- **Architecture Differences**: BDA handles section boundaries automatically
+- **Alternative Workflows**: Available options like "View/Edit Data", Configuration updates, and document reprocessing
+- **Future Considerations**: Guidance on using Pattern-2/Pattern-3 for fine-grained section control
+
+
 ## Document Analytics
 
 The Document Analytics feature allows users to query their processed documents using natural language and receive results in various formats including charts, tables, and text responses.

iam-roles/cloudformation-management/IDP-Cloudformation-Service-Role.yaml

@@ -13,7 +13,7 @@ Resources:
   CloudFormationServiceRole:
     Type: AWS::IAM::Role
     Properties:
-      RoleName: IDPAcceleratorCloudFormationServiceRole
+      RoleName: !Sub '${AWS::StackName}-CFServiceRole'
       AssumeRolePolicyDocument:
         Version: '2012-10-17'
         Statement:
@@ -109,7 +109,7 @@ Resources:
   PassRolePolicy:
     Type: AWS::IAM::ManagedPolicy
     Properties:
-      ManagedPolicyName: IDP-PassRolePolicy
+      ManagedPolicyName: !Sub '${AWS::StackName}-PassRolePolicy'
       Description: Policy to allow passing the IDP CloudFormation service role
       PolicyDocument:
         Version: '2012-10-17'