Skip to content

Conversation

@abhraina-aws
Copy link
Contributor

PR having end to end tests for testing qcli

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

abhraina-aws and others added 2 commits November 5, 2025 05:48
* feat: Initial Framework with working tests

* QCLI Automation for Basic chat

* QCLI automation for Compact,MCP,Usage,Hooks,Subscribe,Model tests

* Q CLI Automation for context and help

* Q-CLI automated  testcases for Agent commands

* feat: added a better way to run tests categorized by features

* fix: added an updated readme

* Q CLI Automation for basic chat and context

* QCLI Automation for Tools,Issue,Promt tests

* Q CLI Automation for MCP

* QCLI automation prompt list command

* QCLI Automation for MCP

* QCLI test case automation for dynamic model selection and bug fix for MCP commands

* QCLI Automation for MCP tests

* QCLI Automation for tools,model,editor,subscribe,usage,compact,hooks,issues,tests

* QCLI Test automation - merged agent feature test into one

* QCLI Test automation - merged context feature test into one

* QCLI Test automation - seperated common methods

* QCLI Test automation - merged ai_prompts feature test into one

* QCLI Test automation - merged agent feature test into one

* QCLI Automation for MCP Feature

* QCLI Automation for hooks,usage,issue,tools,subscribe,model,compact,editor tests

* Added some additional scripts for specific runs

* Refactor: Merge save/load into one file, remove verbose from MCP context

* Remove failing MCP test case

* Created folder structure for e2e test

* Fixed the warnings in code

* Modified the tests features to support multiple features

* QCLI Automation for core_session,integration,mcp,model,session_mgmt tests

* Reduce read_response timeout from 20s to 4s

* Refactor: Remove verbose from MCP context and save/load and fixed the failing test case for adding multiple file as context.

* Migrated to python script with HTML report generation

* Add new tests cases test_add_and_remove_mcp_command and test_q_mcp_status_command

* added html template

* Test wise console output

* Test wise console output for single thread

* Move reports to reports folder

* Script updated to support test description

* Added description to  agent and usage command tests , add .gitignore

* Fixed  individual test output

* Removed all regression test cases

* Added descriptions to tests

* Modfied system info

* Fixed the Suite tital to capitalize

* Clean console output

* Removed the tiles from summary section , added table in HTML report

* Added histogram for the test pass fail

* Validate the user input for passed features

* Fixed the summary cards layout

* Sending prompts used execute command instead of send prompts

* Fixed graph hover text label

* Added highlights on failed features

* QCLI Automation for tools

* move common functions from lib to individual test files to fix intermittent /quit issue, fix MCP failures, and add compact feature tests

* add compact new tests and fix chat function for help,clear,ai_prompts tests

* Add new test cases related to tools and fix the warning

* Updated help section of script

* Added executng animation while executing cargo command

* Improved quiet mode

* Add detail descriptions to 5 test cases related to tools.

* add new tests for editor feature

* Fixed quiet mode

* Fixed quiet mode with logs

* add new tests for editor,compact feature and description for editor , comoact feature test cases

* Fixed order of features

* Add Q subcommand support and corresponding tests

- Introduced `q_subcommand` feature in Cargo.toml
- Implemented functions for starting and executing Q subcommands in lib.rs
- Added tests for `q chat`, `q doctor`, and `q translate` commands
- Updated all_tests.rs to include the new q_subcommand tests

* added new test case for compact feature

* Improved animation

* Add regression tests for MCP feature

* Modified the report css for code

* Refactor Q CLI subcommand execution and enhance tests

- Replaced `start_q_subcommand` and `read_session_response` with `execute_q_subcommand` and `execute_q_subcommand_with_stdin` for improved command execution and response handling.
- Added new tests for `q mcp` subcommands, verifying help commands and functionality.
- Enhanced existing test descriptions for clarity and consistency.

* Add tests for q settings and q whoami subcommands

* Add tests for q debug and q inline subcommands

* Refactor test descriptions to use HTML code tags for command formatting

- Updated test output messages in various test files to wrap command names in <code> tags for better readability.

* Enhance MCP subcommand tests and add q debug build related tests

* Add help subcommand tests for q inline commands and Add todos command support and enhance related tests

* Updated readme file , wrokspace clean up

* Add tests for /todos resume and /todos delete commands, enhance existing tests

* Added tests for q whoami subcommand and fixed failing tests for features agebt,integarion and q-subcommand

* add 3 tests for experiment commands feature

* Increase time limit to improve command response handling and prevent premature termination

* Add changelog command and experiment feature tests

* Added 3 tests under q_subcommand feature

* Refactor test files to utilize centralized chat session management
- Removed redundant chat session  code from individual test files.

* added 2 tests for q_subcommand

* QCLI e2etesting - fixed warnings

* feat(knowledge): implement pattern filtering and settings integration (TI-3, TI-5) (#2545)

- Add pattern filtering foundation with --include/--exclude CLI flags (TI-3)
- Implement knowledge settings integration with database system (TI-5)
- Add comprehensive pattern filter module with glob-style support
- Enhance file processor with async pattern filtering capabilities
- Add extensive test coverage for pattern filtering functionality
- Update knowledge CLI with improved error handling and validation
- Add settings support for chunk size, overlap, and file limits

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* fix:  Add client-side modelName mapping for backward compatibility  (#2604)

* add mapping

* adjust import location

* compare name first

* feat: add delay tracking interceptor for retry notifications (#2607)

* fix(agent): tool permission override (#2606)

* fix: add more safety checks for execute_bash readonly checks (#2605)

* fix: change fallback (#2615)

* chore: bump version to 1.14.0 (#2616)

* Revert "fix(agent): tool permission override (#2606)" (#2618)

This reverts commit dfaa580.

* Knowledge beta improvements phase 2: Refactor async_client and add support for BM25 (#2608)

* [feat] Adds support to embedding-type

- Remove unused BM25TextEmbedder from embedder factory
- Replace with MockTextEmbedder for Fast embedding type
- Remove bm25.rs file and related imports/exports
- Fix BM25Context and SemanticContext to save data after adding points
- Fix BM25 data filename from bm25_data.json to data.bm25.json
- Add base_dir storage to ContextManager for proper path resolution
- Major refactoring to async context management with background operations
- Adds separate optimized index for bm25
- Fix all clippy warnings and remove dead code

BM25 search now works correctly with persistent contexts.

* fix: Update cancel_most_recent_operation to use OperationManager

- Fix cancel_most_recent_operation to delegate to OperationManager instead of accessing active_operations directly
- Add missing cancel_most_recent_operation method to OperationManager
- Ensures proper separation of concerns in the refactored architecture

* fix: Remove BM25 from benchmark tests

- Remove BM25TextEmbedder references from benchmark_test.rs
- Remove benchmark_bm25_model function
- Keep only Candle model benchmarks
- Fixes compilation error after BM25TextEmbedder removal

* docs: Update semantic-search-client README for index types

- Update description to mention BM25 and vector embeddings
- Add Multiple Index Types feature
- Update Embeddings section to Index Types section
- Remove ONNX references (no longer supported)
- Reflect Fast (BM25) vs Best (Semantic) terminology
- Update section headers for consistency

* fix: remove auto-save from context add_data_points methods

- Remove automatic save() calls from add_data_points in both semantic_context.rs and bm25_context.rs
- Add explicit save() calls in context_creator.rs after data addition is complete
- Improves performance by avoiding multiple disk writes during batch operations
- Addresses PR #2608 feedback about inefficient disk I/O on each context addition

* fix: resolve compilation error and operation cancel warnings

- Fix return type mismatch in knowledge_store.rs cancel_operation method
- Change cancel_most_recent_operation to return Ok instead of Err when no operations exist
- Eliminates 'Failed to cancel operations' warnings when no operations are active

* fix: improve error handling and code cleanup

- Update error handling in knowledge_store.rs
- Clean up context_creator.rs formatting and comments

---------

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* fix: use_aws printing default profile when its not used, minor updates to agent docs (#2617)

* chore(models): change fallback model, align with all clients (#2624)

* feat: add github action for release notification (#2625)

* feat: add github action for release notification

* feat(agent): hot swap (#2637)

* changes prompt list result to be sent over via messenger

* changes tool manager orchestrator tasks to keep prompts

* changes mpsc to broadcast

* restores prompt list functionality

* restore prompt get functionality

* adds api on tool manager to hotswap

* spawns task to send deinit msg via messenger

* adds slash command to hotswap agent

* modifies load tool wait time depending on context

* adds comments to retry logic for prompt completer

* fixes lint

* adds pid field to messenger message

* adds interactive menu for swapping agent

* fixes stale mcp load record

* documents build method on tool manager builder and refactor to make the build method smaller

* feat: Implement wildcard pattern matching for agent allowedTools (#2612)

- Add globset-based pattern matching to support wildcards (* and ?) in allowedTools
- Create util/pattern_matching.rs module with matches_any_pattern function
- Update all native tools (fs_read, fs_write, execute_bash, use_aws, knowledge) to use pattern matching
- Update MCP custom tools to support wildcard patterns while preserving exact server-level matching
- Standardize imports across tool files for consistency
- Maintain backward compatibility with existing exact-match behavior

Enables agent configs like:
- "fs_*" matches fs_read, fs_write
- "@mcp-server/tool_*" matches tool_read, tool_write
- "execute_*" matches execute_bash, execute_cmd

* fixes unwrap on pid (#2657)

* ci fix (#2658)

* feat: added mcp admin level configuration with GetProfile (#2639)

* first pass

* add notification when /mcp & /tools

* clear all tool related filed in agent

* store mcp_enabled in chatsession & conversationstate

* delete duplicate api call

* set mcp_enabled value after load

* remove clear mcp configs method

* clippy

* remain@builtin/ and *, add a ut for clear mcp config

* automated agent edit test case.

* reduced timeout duration and added test-agent-edit

* added one test case  for settings format

* added changes for mod.rs

* code fix

* fix(agent): tool permission (#2619)

* adds warnings for when tool settings are overridden by allowed tools

* adjusts tool settings eval order

* modifies doc

* moves warning to be displayed after splash screen

* canonicalizes paths prior to making glob sets

* simplifies overridden warning message printing logic

* adds more doc on path globbing

* chore: bumps version to 1.14.1 (#2662)

* adding telemetry for mcp tool names available and mcp tool names selected (#2655)

* feat: add tangent mode for context isolated conversations (#2634)

- Add /tangent command to toggle between normal and tangent modes
- Preserve conversation history during tangent for context
- Restore main conversation when exiting tangent mode
- Add Ctrl+T keyboard shortcut for quick access
- Visual indicator: green [T] prompt in tangent mode
- Comprehensive tests for state management and UI components

Allows users to explore side topics without polluting main conversation context.

* Apply cargo +nightly fmt formatting (#2664)

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* feat: implement agent-scoped knowledge base and  context-specific search (#2647)

* Short-Term fix for SendTelemetry API Validation errors (#2694)

* feat: add introspect tool for Q CLI self-awareness (#2677)

- Add introspect tool with comprehensive Q CLI documentation
- Include auto-tangent mode for isolated introspect conversations
- Add GitHub links for documentation references
- Support agent file locations and built-in tools documentation
- Add automatic settings documentation with native enum descriptions
- Use strum EnumMessage for maintainable setting descriptions

* Format fix (#2697)

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* Update q-developer smithy clients and reformat files (#2698)

* Update q-developer smithy clients and reformat files

* Fix formatting

* feat: add tangent mode duration tracking telemetry (#2710)

- Track time spent in tangent mode sessions
- Send duration metric when exiting tangent mode via /tangent command
- Use codewhispererterminal_chatSlashCommandExecuted metric with:
  - command: 'tangent'
  - subcommand: 'exit'
  - value: duration in seconds
- Add comprehensive tests for duration calculation
- Enables analytics on tangent mode usage patterns

* feat: add to-do list functionality to QCLI (#2533)

* first round changes for agent generate for workshopping (#2690)

* adding in agent contribution metric

* chore: fix lints

* agent generate where user is prompted to include or exclude entire MCP servers

* rebasing changes to newest api for conversation.rs

* making changes according to Brandon's feedback

* clippy

* suppress warning messages

* making changes after code review

* felix feedback changes

---------

Co-authored-by: Xian Wu <xianwwu@amazon.com>
Co-authored-by: Brandon Kiser <brandonkns66@gmail.com>

* fix: update per-prompt timestamp to include local time zone information (#2654)

* feat: add /experiment slash command for toggling experimental features (#2711)

* feat: add /experiment slash command for toggling experimental features

- Add new /experiment command
- Toggle selection interface with colored On/Off status indicators
- Include experiment descriptions and disclaimer about experimental features
- Include documentation so its available in instrospect tool

* Visual apperance fix

* Format fix

---------

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* chore: bump version to 1.15.0 (#2719)

* feat: added similar feature gating of tangent to tangent mode and introspect tool (#2720)

* chore: add /tangent to /experiment (#2721)

* fix: added summary to tangent and showed correct display label for introspect (#2725)

* docs: added the documentation for introspect in the built in tools. (#2727)

* this contains bug fix from the bug bash for agent generate (#2732)

* fixing bugs

* formatting

---------

Co-authored-by: Xian Wu <xianwwu@amazon.com>

* fix: Fixes out of bounds issue with dropdown. (#2726)

* Update retry-interceptor warning message (#2709)

* Add telemetry support for agent contribution tracking (#2699)

- Refactor send_cw_telemetry to handle multiple event types
- Add AgentContribution event handling that sends ChatInteractWithMessageEvent
- Track accepted line count from agent contributions as AgenticCodeAccepted interaction type

* fix: fix to-do list bugs (#2729)

* fix: use abs value of lines added/removed by q-cli (#2737)

* fix: make todo lists an experimental feature (#2740)

* fix: CTRL+C handling during multi-select, auto completion for /agent generate (#2741)

* fixing bugs

* formatting

* fix: CTRL+C handling during multi-select, auto completion for /agent generate

* set use legacy mcp config to false

---------

Co-authored-by: Xian Wu <xianwwu@amazon.com>

* Fix calculation for num-lines contributed by q-cli (#2738)

* Update knowledge base directory path documentation (#2763)

- Changed from ~/.q/knowledge_bases/ to ~/.aws/amazonq/knowledge_bases/
- Default agent uses q_cli_default/ (no alphanumeric suffix)
- Custom agents use <agent-name>_<alphanumeric-code>/ format

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* docs: Update todo list docs for introspect (#2776)

* feat: added tangent & introspect docs & provided to introspect (#2775)

* feat: implement persistent CLI history with file storage (#2769)

- Add Drop trait to InputSource for automatic history saving
- Replace DefaultHistory with FileHistory for persistence
- Store history in ~/.aws/amazonq/cli_history
- Refactor ChatHinter to use rustyline's built-in history search
- Remove manual history tracking in favor of rustyline's implementation
- Add history loading on startup with error handling
- Clean up unused hinter history update methods

* chore: Skip sending profileArn when using custom endpoints (#2777)

* comment profile set

* comment profile in apiclient

* add a helper func

* fix compile issue

* remove dead code tag

* chore(mcp): migrate to rmcp (#2700)

* client struct definition

* clean up unused code

* adds mechanism for checking if server is alive

* prefetches prompts if applicable

* fixes agent swap

fixes agent swap

* only applies process group leader promo for unix

* removes unused import for windows

* renames abstractions for different stages of mcp config

* Dont preserve summary when conversation is cleared (#2793)

* feat: add AGENTS.md to default agent resources (#2812)

- Add file://AGENTS.md to default resources list alongside AmazonQ.md
- Update test to include both AmazonQ.md and AGENTS.md files
- Ensures AGENTS.md is included everywhere AmazonQ.md was previously included

Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>

* feat: add model field support to agent format (#2815)

- Add optional 'model' field to Agent struct for specifying model per agent
- Update JSON schema and documentation with model field usage
- Integrate agent model into model selection priority:
  1. CLI argument (--model)
  2. Agent's model field (new)
  3. User's saved default model
  4. System default model
- Add proper fallback when agent specifies unavailable model
- Extract fallback logic to eliminate code duplication
- Include comprehensive unit tests for model field functionality
- Maintain backward compatibility with existing agent configurations

Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>

* chore: updating doc to surface /agent generate and note block for /knowledge (#2823)

* Properly handle path with trailing slash in file matching (#2817)

* Properly handle path with trailing slash in file matching

Today if a path has a trailing slash, the glob pattern will look like
"/path-to-folder//**" (note the double slash). Glob doesn't work with
double slash actually (it doesn't match anything). As a result, the
permission management for fs_read and fs_write is broken when allowed
or denied path has trailing slash.

The fix is to just manually remove the trailing slash.

* format change

* Fix: Add configurable line wrapping for chat (#2816)

* add a wrapmode in chat args

* add a ut for the wrap arg

* feat(use_aws): add configurable autoAllowReadonly setting (#2828)

- Add auto_allow_readonly field to use_aws Settings struct (defaults to false)
- Update eval_perm method to use auto_allow_readonly setting instead of hardcoded behavior
- Default behavior: all AWS operations require user confirmation (secure by default)
- Opt-in behavior: when autoAllowReadonly=true, read-only operations are auto-approved
- Add comprehensive tests covering all scenarios
- Maintains backward compatibility through configuration

🤖 Assisted by Amazon Q Developer

Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>

* feat: add auto-announcement feature with /changelog command (#2833)

* feat(mcp): enables remote mcp (#2836)

* feat: Add /tangent tail to preserve the last tangent conversation (#2838)

Users can now keep the final question and answer from tangent mode
by using `/tangent tail` instead of `/tangent`. This preserves the last
Q&A pair when returning to the main conversation, making it easy to
retain helpful insights discovered during exploration.

- `/tangent` - exits tangent mode (existing behavior unchanged)
- `/tangent tail` - exits tangent mode but keeps the last Q&A pair

This enables users to safely explore topics without losing the final
valuable insight that could benefit their main conversation flow.

* feat: add daily heartbeat telemetry (#2839)

Tracks daily active users by sending amazonqcli_dailyHeartbeat
event once per day. Uses fail-closed logic to prevent spam
during database errors.

* fix: update dangerous patterns for execute bash to include $ (#2811)

* docs: fix local agent directory path in documentation (#2749)

* docs: fix local agent directory path

- Fix local agent path from .aws/amazonq/cli-agents/ to .amazonq/cli-agents/
- Global paths (~/.aws/amazonq/cli-agents/) remain correct
- Aligns documentation with source code implementation

* fix: correct workspace agent path in /agent help message

The help message for the /agent command incorrectly showed the workspace
agent path as 'cwd/.aws/amazonq/cli-agents' when it should be
'cwd/.amazonq/cli-agents' (without the .aws directory).

This fix aligns the help text with the actual WORKSPACE_AGENT_DIR_RELATIVE
constant defined in directories.rs.

* Invalid pointer to trace log location (#2734)

* Fix bug README.md (#2569)

* fix: Layout fix (#2798)

* docs: Update experiment docs to contain todo lists (#2791)

* feat: Add support for comma-containing arguments in MCP --args parameter (#2754)

* chore: add extra curl flags for debugging build during feed.json failures (#2843)

* fix: remove downloading feed during build (#2844)

* feat(execute_bash): change autoAllowReadonly default to false for security (#2846)

- Change default_allow_read_only() from true to false for secure by default behavior
- Default behavior: all bash commands require user confirmation (secure by default)
- Opt-in behavior: when autoAllowReadonly=true, read-only commands are auto-approved
- Use autoAllowReadonly casing to match use_aws tool pattern
- Update documentation to reflect new default value and consistent naming
- Add comprehensive tests covering all scenarios
- Maintains backward compatibility through configuration
- Follows same pattern as use_aws autoAllowReadonly setting

🤖 Assisted by Amazon Q Developer

Co-authored-by: Matt Lee <mrlee@amazon.com>

* feat(agent): add edit subcommand to modify existing agents (#2845)

- Add Edit subcommand to AgentSubcommands enum
- Implement edit functionality that opens existing agent files in editor
- Use Agent::get_agent_by_name to locate and load existing agents
- Include post-edit validation to ensure JSON remains valid
- Add comprehensive tests for the new edit subcommand
- Support both --name and -n flags for agent name specification

🤖 Assisted by Amazon Q Developer

Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>

* fix(mcp): not being able to refresh tokens for remote mcp (#2849)

* adds registration persistance for token refresh

* truncates on tool description

* Modifies oauth success message

* adds time stamps on mcp logs

* fix(agent): add edit subcommand support to /agent slash command (#2854)

- Add Edit variant to AgentSubcommand enum in profile.rs
- Implement edit functionality for slash command usage
- Use Agent::get_agent_by_name to locate existing agents
- Include post-edit validation and agent reloading
- Add edit case to name() method for proper command routing
- Enables /agent edit --name <agent> usage in chat sessions

🤖 Assisted by Amazon Q Developer

Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>

* fixes creation of cache directory panic when path already exists (#2857)

* Reduce default fs_read trust permission to current working directory only (#2824)

* Reduce default fs_read trust permission to current working directory only

Previously by default fs_read is trusted to read any file on user's file
system. This PR reduces the fs_read permission to CWD only. This means user
can still access any file under CWD without prompt. But if user needs to
access file outside CWD, she will be prompted for explicit approval.

User can still explicitly add fs_read to trusted tools in chat / agent
definition so fs_read can read any file without prompt.

This change essentially adds a layer of defense against prompt injection
by following the least-privilege principle.

* remove allow_read_only since it is always false now

* fixes remote mcp creds not being written when obtained (#2878)

* fixes bug where refreshed credentials gets deleted (#2879)

* fix(mcp): bug where refreshed credentials gets deleted (#2880)

* Update bm25 to v2.3.2 and ignore unmaintained fxhash advisory (#2872)

- Updated bm25 from v2.2.1 to v2.3.2 (latest version)
- Added RUSTSEC-2025-0057 to deny.toml ignore list for unmaintained fxhash
- fxhash is a transitive dependency of bm25, waiting for upstream migration to rustc-hash

Co-authored-by: Kenneth S. <kennvene@amazon.com>

* adds temp message to still loading section of /tools (#2881)

* chore:  removes codeowner for schema (#2892)

* chore(agent): updates agent config schema (#2891)

* chore: version bump (#2894)

* fix format (#2895)

* fix(mcp): shell expansion not being done for stdio command (#2915)

* Bump up version for hotfix (#2916)

* chore: add 1.16.1 feed.json entry (#2919)

* Change autocomplete shortcut from ctrl-f to ctrl-g (#2825)

* Change autocomplete shortcut from ctrl-f to ctrl-g

The reason is ctrl-f is the standard shortcut in UNIX for moving cursor
forward by 1 character. You can find it being supported everywhere...
in your browser, your terminal, etc.

* make the autocompletion key configurable

* feat: add support for preToolUse and postToolUse hook (#2875)

* fix(mcp): oauth issues (#2925)

* fix incorrect scope for mcp oauth

* reverts custom tool config enum change

* fixes display task overriding sign in notice

* updates schema

* fix(chat): reset pending tool state when clearing conversation (#2855)

Reset tool_uses, pending_tool_index, and tool_turn_start_time to prevent
orphaned tool approval prompts after conversation history is cleared.

Co-authored-by: Niraj Chowdhary <chownira@amazon.com>

* Trim region to avoid login failures (#2930)

* chore: copy change for warning message for oauth redirect page (#2931)

* fix: removes deny unknown fields in mcp config (#2935)

* Normalize and expand relative-paths to absolute paths (#2933)

* Bump version to 1.16.2 and update feed.json (#2938)

* changes how mcp command gets expanded (#2940)

* Update default tool label for execute bash (#2945)

* fix: incorrect wrapping for response text (#2900)

* Improve error messages for dispatch failures (#2969)

* automated tangent and agent generate command.

* automated introspect command and handle wanrnings.

* added code to automate /prompts get command

* fixed the html rendering issue.

* automated whoami command and handle html color rendering issue.

* Modified histogram bar labels

* Added support for custom timeout

* added multiline test case automation and handle warning

* automated ctrl+s command and refactor ctrl+j command for multiline.

* added test case automation support for /agent swap

* added test case automation for alt+enter and added timeout to existing function for long execution.

* Added python script to analyse the qcli e2e test performances

* Changes for improving test case execution.

* Q CLi e2e testing updated  python script to analyse the qcli e2e test performances

* fixed failed test cases because of latest q cli version and remove all warnings.

* Q Cli e2e testing automation report performance review dashboard modified.

* code changes for /agent schema test case automation

* code changes to automate  q settings delete command

* added code to automate q quit subcommand.

* added code to automate /todos clear-finished command.

* Q cli e2e tests fix warnings

---------

Co-authored-by: abhraina-aws <abhraina@amazon.com>
Co-authored-by: Shreya [C] Bhagat <shrebhaa@amazon.com>
Co-authored-by: sayema Anjum <sayemanc@amazon.com>
Co-authored-by: Kenneth Sanchez V <kensave@hotmail.com>
Co-authored-by: Kenneth S. <kennvene@amazon.com>
Co-authored-by: evanliu048 <evannliu@amazon.com>
Co-authored-by: kkashilk <93673379+kkashilk@users.noreply.github.com>
Co-authored-by: Felix Ding <dingfeli@amazon.com>
Co-authored-by: Brandon Kiser <51934408+brandonskiser@users.noreply.github.com>
Co-authored-by: Nitish [C] Dhok <gnitesdh@amazon.com>
Co-authored-by: xianwwu <xianwu92@gmail.com>
Co-authored-by: kiran-garre <137448023+kiran-garre@users.noreply.github.com>
Co-authored-by: Xian Wu <xianwwu@amazon.com>
Co-authored-by: Brandon Kiser <brandonkns66@gmail.com>
Co-authored-by: Justin Moser <jmmoser@users.noreply.github.com>
Co-authored-by: Matt Lee <1302416+mr-lee@users.noreply.github.com>
Co-authored-by: Matt Lee <mr-lee@users.noreply.github.com>
Co-authored-by: Erben Mo <erbenmo@gmail.com>
Co-authored-by: yayami3 <116920988+yayami3@users.noreply.github.com>
Co-authored-by: Bart van Bragt <github.com@vanbragt.com>
Co-authored-by: Ennio Pastore <ennio1991@gmail.com>
Co-authored-by: Michael Orlov <34108460+harleylrn@users.noreply.github.com>
Co-authored-by: Matt Lee <mrlee@amazon.com>
Co-authored-by: Erben Mo <moerben@amazon.com>
Co-authored-by: nirajchowdhary <226941436+nirajchowdhary@users.noreply.github.com>
Co-authored-by: Niraj Chowdhary <chownira@amazon.com>
.find(|line| !line.starts_with("!") && !line.starts_with(">") && !line.trim().is_empty())
.expect("Failed to get username from whoami command")
.trim();
println!("✅ Current username: {}", username);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
println!("✅ Current username: {}", username);

let agent_path = format!("/Users/{}/.aws/amazonq/cli-agents/{}.json", username, agent_name);
println!("✅ Agent path: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.

if std::path::Path::new(&agent_path).exists() {
std::fs::remove_file(&agent_path)?;
println!("✅ Agent file deleted: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
std::fs::remove_file(&agent_path)?;
println!("✅ Agent file deleted: {}", agent_path);
} else {
println!("⚠️ Agent file not found at: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
.find(|line| !line.starts_with("!") && !line.starts_with(">") && !line.trim().is_empty())
.expect("Failed to get username from whoami command")
.trim();
println!("✅ Current username: {}", username);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
println!("✅ Current username: {}", username);

let agent_path = format!("/Users/{}/.aws/amazonq/cli-agents/{}.json", username, agent_name);
println!("✅ Agent path: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.

if std::path::Path::new(&agent_path).exists() {
std::fs::remove_file(&agent_path)?;
println!("✅ Agent file deleted: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
std::fs::remove_file(&agent_path)?;
println!("✅ Agent file deleted: {}", agent_path);
} else {
println!("⚠️ Agent file not found at: {}", agent_path);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
username
to a log file.
// Count lines with "trusted" to ensure multiple tools are trusted
let trusted_count = tools_response.matches("trusted").count();
assert!(trusted_count > 0, "No trusted tools found");
println!("✅ Found {} instances of 'trusted' in tools list", trusted_count);

Check failure

Code scanning / CodeQL

Cleartext logging of sensitive information High test

This operation writes
trusted_count
to a log file.
}

assert!(!std::path::Path::new(&agent_path).exists(), "Agent file should be deleted");
println!("✅ Agent deletion verified");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are mixing the deletion with the actual test here. I am we are manually deleting these and then using it as a validation.


#[test]
#[cfg(all(feature = "ai_prompts", feature = "sanity"))]
fn test_prompts_list_command() -> Result<(), Box<dyn std::error::Error>> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this and the one below testing the same thing?
Also, there are no prompts here to list. We should add some and test the functionality properly.

@kensave
Copy link
Contributor

kensave commented Nov 6, 2025

Do we have any metrics on the coverage of this suite of tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants