Introduce hook and context management to OpenSearch Agents

[mingshl]

Description

Add Complete Context Management Framework with Memory-Integrated Hook Events to ML Agents

This PR introduces a comprehensive context management system for ML agents, featuring memory-integrated hook
events, complete hook registry with fault-tolerant execution, enhanced context integration, and intelligent
resource management for enterprise-grade deployments.

Major Features:

1. Complete Hook Registry System
   • Fault-Tolerant Event Emission: emitEvent() with exception isolation and comprehensive logging
   • Thread-Safe Operations: ConcurrentHashMap ensuring concurrent callback registration and execution
   • Callback Management: Type-safe callback registration with monitoring capabilities
   • Production-Grade Error Handling: Individual callback failures don't affect system operation

2. Comprehensive Hook Event Architecture
   • HookEvent Base Class: Abstract base with invocation state for all lifecycle events
   • PreLLMEvent: pre llm event with context integration and retrieved re-act interactions, including tool used result/ completed steps
   • EnhancedPostToolEvent: Tool event with context integration
   • Event Hierarchy: Well-designed inheritance from HookEvent to specialized events

3. Complete CRUD REST API Implementation
   • Create Operations: MLCreateContextManagementTemplateAction, MLCreateContextManagementTemplateRequest,
   MLCreateContextManagementTemplateResponse
   • Read Operations: MLGetContextManagementTemplateAction, MLGetContextManagementTemplateRequest,
   MLGetContextManagementTemplateResponse
   • Delete Operations: MLDeleteContextManagementTemplateAction, MLDeleteContextManagementTemplateRequest,
   MLDeleteContextManagementTemplateResponse
   • Full CRUD Lifecycle: Complete template management with create, read, and delete operations

4. Context Management Integration
   • ML Agent Integration: contextManagementName and contextManagement fields with validation
   • Template System: ContextManagementTemplate with hooks, validation, and serialization
   • Hook Provider Integration: ContextManagerHookProvider registers callbacks for all event types

5. Rich Context State Management
   • ContextManagerContext: Comprehensive context with prompts, history, tools, interactions, and parameters
   • Utility Methods: getEstimatedTokenCount(), getMessageCount(), addParameter(), addToolInteraction()
   • Memory Integration: Context managers can access both current context and retrieved memory
   • Dynamic Modification: Runtime context updates with memory and invocation state tracking

6. Event-Driven Execution Pipeline
   • Tool Event Processing: handlePostTool() processes enhanced tool events with context
   • Pre-LLM Processing: handlePreLLM() processes context before LLM execution
   • Memory Event Processing: handlePostMemory() processes memory events with context and history
   • Automatic Execution: All events processed automatically through hook registry

7. Complete Activation Rules System
   • TokensExceedRule: Context-aware activation using comprehensive token estimation
   • MessageCountExceedRule: Conversation length management with memory integration
   • CompositeActivationRule: Complex conditions considering memory and context state

8. Token Counting Architecture
   • TokenCounter Interface: Standardized interface with multiple truncation strategies
   • Context Integration: Token counters integrated with activation rules and memory state
   • Pluggable Design: Interface supports model-specific implementations

9. Pluggable Context Manager Architecture
   • ContextManagerConfig: Configuration with type, activation, and manager settings
   • Template Management: Complete template system with memory-integrated hooks
   • Registry Integration: Managers registered for memory, tool, and LLM events

Technical Highlights:
• Comprehensive Event Model: Complete lifecycle coverage with memory, tool, and LLM events
• Fault-Tolerant Processing: Exception isolation ensures system stability across all event types
• Rich Context Access: Context managers have access to full context and memory

Architecture Benefits:
• Complete Lifecycle Coverage: Events cover all major phases of ML agent execution
• Integrated Memory Management: Memory retrieval and context management work together seamlessly
• Enterprise Reliability: Fault-tolerant design with comprehensive error handling

Use Cases:
• Comprehensive Resource Management: Manage tokens across current context and memory history

Impact:
• Comprehensive Agent Lifecycle: Full coverage of memory, tool, and LLM execution phases
• Advanced AI Orchestration: Sophisticated context management with memory integration

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

callbacks.computeIfAbsent --> this is not thread-safe. Do you see any concern?

[mingshl]

@Zhangxunmt can you explain your intention please?

[Zhangxunmt]

I don't see any new Threads added in this PR. So iirc there's no multiple threads in agents registration which is the only trigger to add hooks. If this addCallback is never used in the agent execution we should be good. Please double check @mingshl .