|
1 | 1 | # Test Case Enhancement Rules |
2 | 2 |
|
3 | | -## Assistant Workflow for Adding Comprehensive Test Cases |
| 3 | +## Simple Enhancement Workflow |
4 | 4 |
|
5 | | -### Automated Workflow (Recommended) |
| 5 | +When user requests test case enhancement: |
6 | 6 |
|
7 | | -For single problem enhancement: |
| 7 | +### 1. Problem Resolution |
8 | 8 |
|
9 | | -1. **Use automation script**: `./.amazonq/rules/test-case-enhancement/scripts/enhance_single_problem.sh <problem_name>` |
10 | | -2. **Verify results**: `./.amazonq/rules/test-case-enhancement/scripts/verify_enhancement.sh <problem_name>` |
| 9 | +- Use active file context or user-provided problem name |
| 10 | +- If unclear, run: `poetry run python .templates/check_test_cases.py --threshold=10 --max=1` |
11 | 11 |
|
12 | | -For batch enhancement: |
13 | | - |
14 | | -1. **Find problems**: `./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh` |
15 | | -2. **Batch enhance**: `./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh` |
16 | | - |
17 | | -### Manual Workflow (Fallback) |
18 | | - |
19 | | -When user requests to enhance test cases for a problem, the assistant will: |
20 | | - |
21 | | -### 1. Problem Resolution (Priority Order) |
22 | | - |
23 | | -- **FIRST**: Try to resolve from context - check active file path or user-provided problem name |
24 | | -- **SECOND**: If context resolution fails, THEN run `poetry run python .templates/check_test_cases.py --threshold=10 --max=1` to auto-detect 1 problem with <10 test cases |
25 | | -- **LAST**: If both above fail, ask user to explicitly specify problem name |
26 | | - |
27 | | -### 2. Test Case Generation |
28 | | - |
29 | | -- Read `leetcode/{problem_name}/README.md` for problem understanding |
30 | | -- Analyze existing test cases in `leetcode/{problem_name}/tests.py` |
31 | | -- Generate comprehensive test cases covering: |
32 | | - - **Edge cases**: Empty inputs, single elements, boundary values |
33 | | - - **Corner cases**: Maximum/minimum constraints, special patterns |
34 | | - - **Normal cases**: Typical scenarios with varied complexity |
35 | | - - **Error cases**: Invalid inputs (if applicable) |
36 | | - |
37 | | -### 3. Initial Validation |
38 | | - |
39 | | -- Run `make p-test PROBLEM={problem_name}` to verify current implementation |
40 | | -- **If errors found**: |
41 | | - - DO NOT update implementation automatically |
42 | | - - Only update test cases if they're incorrect |
43 | | - - If implementation seems wrong, ASK USER first before modifying |
44 | | - |
45 | | -### 4. JSON Template Update |
46 | | - |
47 | | -- Update corresponding `.templates/leetcode/json/{problem_name}.json` |
48 | | -- Add new test cases to `test_cases` field in proper format |
49 | | -- Maintain existing test structure and naming conventions |
50 | | - |
51 | | -### 5. Backup and Regeneration Process |
52 | | - |
53 | | -- **Backup**: Move `leetcode/{problem_name}/` to `.cache/leetcode/{problem_name}/` |
54 | | -- **Regenerate**: Run `make p-gen PROBLEM={problem_name} FORCE=1` |
55 | | -- **Lint check**: Run `make p-lint PROBLEM={problem_name}` |
56 | | -- **Iterate**: If lint fails, update JSON and regenerate until passes |
57 | | - |
58 | | -### 6. Solution Preservation |
59 | | - |
60 | | -- Copy `solution.py` from backup to newly generated structure |
61 | | -- Run `make p-test PROBLEM={problem_name}` to verify tests pass |
62 | | -- **If tests fail**: Go back to step 4, update JSON, and iterate until passes |
63 | | - |
64 | | -### 7. Cleanup and Restore |
65 | | - |
66 | | -- **CRITICAL**: Remove entire newly generated `leetcode/{problem_name}/` directory |
67 | | -- **CRITICAL**: Restore original structure from `.cache/leetcode/{problem_name}/` backup |
68 | | -- **CRITICAL**: Only THEN copy enhanced `test_solution.py` from generated files to restored structure |
69 | | -- **CRITICAL**: Preserve existing solution class parametrization - if original test had multiple solution classes, restore them |
70 | | -- Verify final state with `make p-test PROBLEM={problem_name}` |
71 | | -- Clean up backup directory after successful verification |
72 | | - |
73 | | -## Test Case Quality Standards |
74 | | - |
75 | | -### Coverage Requirements |
76 | | - |
77 | | -- **Minimum 10 test cases** per problem |
78 | | -- **Total test cases after enhancement must exceed the threshold** used for detection (e.g., if threshold=10, final count must be >10) |
79 | | -- **Edge cases**: 20-30% of total test cases |
80 | | -- **Normal cases**: 50-60% of total test cases |
81 | | -- **Corner cases**: 20-30% of total test cases |
82 | | - |
83 | | -### Test Case Categories |
84 | | - |
85 | | -#### Edge Cases |
86 | | - |
87 | | -- Empty inputs: `[]`, `""`, `None` |
88 | | -- Single element: `[1]`, `"a"` |
89 | | -- Boundary values: `[0]`, `[1]`, `[-1]` |
90 | | -- Maximum/minimum constraints from problem description |
91 | | - |
92 | | -#### Corner Cases |
93 | | - |
94 | | -- Duplicate elements: `[1,1,1]` |
95 | | -- Sorted/reverse sorted arrays: `[1,2,3]`, `[3,2,1]` |
96 | | -- All same elements: `[5,5,5,5]` |
97 | | -- Alternating patterns: `[1,0,1,0]` |
98 | | - |
99 | | -#### Normal Cases |
100 | | - |
101 | | -- Mixed positive/negative numbers |
102 | | -- Various array sizes within constraints |
103 | | -- Different data patterns and structures |
104 | | -- Representative problem scenarios |
105 | | - |
106 | | -### JSON Format Requirements |
107 | | - |
108 | | -- Use single quotes for Python strings in test cases |
109 | | -- Follow existing parametrize format |
110 | | -- Maintain type hints in parametrize_typed |
111 | | -- Ensure test_cases string is valid Python list syntax |
112 | | -- **NEVER include custom solution classes** in test_imports - only import the main solution class specified in solution_class_name |
113 | | -- **PRESERVE existing solution class parametrization** - if original test had multiple solution classes, restore them after JSON regeneration |
114 | | - |
115 | | -## Automation Scripts 🤖 |
116 | | - |
117 | | -**RECOMMENDED APPROACH**: Use prebuilt scripts in `.amazonq/rules/test-case-enhancement/scripts/` to streamline the enhancement process: |
118 | | - |
119 | | -### Find Problems Needing Enhancement |
| 12 | +### 2. Enhancement Process |
120 | 13 |
|
121 | 14 | ```bash |
122 | | -# Find problems with ≤10 test cases (default) |
123 | | -./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh |
124 | | - |
125 | | -# Custom threshold and max results |
126 | | -./.amazonq/rules/test-case-enhancement/scripts/find_problems_needing_enhancement.sh 8 15 |
| 15 | +# Simple 4-step process: |
| 16 | +# 1. Update JSON template with more test cases (12-15 total) |
| 17 | +# 2. Backup original |
| 18 | +mv leetcode/{problem_name} .cache/leetcode/{problem_name} |
| 19 | +# 3. Regenerate with enhanced tests |
| 20 | +make p-gen PROBLEM={problem_name} FORCE=1 && make p-lint PROBLEM={problem_name} |
| 21 | +# 4. Restore original solution, keep enhanced tests |
| 22 | +cp .cache/leetcode/{problem_name}/solution.py leetcode/{problem_name}/solution.py |
127 | 23 | ``` |
128 | 24 |
|
129 | | -### Enhance Single Problem |
| 25 | +### 3. Verification |
| 26 | + |
| 27 | +- Run `make p-test PROBLEM={problem_name}` |
| 28 | +- Fix any incorrect expected values in test cases |
| 29 | +- Update JSON template with corrections |
| 30 | + |
| 31 | +### 4. Restore Backup |
130 | 32 |
|
131 | 33 | ```bash |
132 | | -# Enhance specific problem (handles full workflow automatically) |
133 | | -./.amazonq/rules/test-case-enhancement/scripts/enhance_single_problem.sh <problem_name> |
| 34 | +# Copy enhanced test_solution.py to backup |
| 35 | +cp leetcode/{problem_name}/test_solution.py .cache/leetcode/{problem_name}/ |
| 36 | +# Restore all original files (preserves user edits) |
| 37 | +rm -rf leetcode/{problem_name} |
| 38 | +mv .cache/leetcode/{problem_name} leetcode/{problem_name} |
134 | 39 | ``` |
135 | 40 |
|
136 | | -### Batch Enhancement |
| 41 | +## Test Case Standards |
137 | 42 |
|
138 | | -```bash |
139 | | -# Enhance up to 10 problems with ≤10 test cases (default) |
140 | | -./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh |
| 43 | +### Coverage Requirements |
141 | 44 |
|
142 | | -# Custom batch size and threshold |
143 | | -./.amazonq/rules/test-case-enhancement/scripts/enhance_batch_problems.sh 5 8 |
144 | | -``` |
| 45 | +- **Minimum 12 test cases** per problem |
| 46 | +- **Edge cases**: Empty inputs, single elements, boundary values |
| 47 | +- **Corner cases**: Maximum/minimum constraints, duplicates, sorted arrays |
| 48 | +- **Normal cases**: Mixed scenarios with varied complexity |
145 | 49 |
|
146 | | -### Verify Enhancement Results |
| 50 | +### JSON Format |
147 | 51 |
|
148 | | -```bash |
149 | | -# Verify enhancement was successful |
150 | | -./.amazonq/rules/test-case-enhancement/scripts/verify_enhancement.sh <problem_name> |
151 | | -``` |
| 52 | +- Use single quotes for Python strings: `'hello'` not `"hello"` |
| 53 | +- Follow existing parametrize format |
| 54 | +- Ensure valid Python list syntax in test_cases field |
152 | 55 |
|
153 | | -## Manual Commands Reference |
| 56 | +## Quick Commands |
154 | 57 |
|
155 | 58 | ```bash |
156 | | -# Find problems needing more test cases |
157 | | -poetry run python .templates/check_test_cases.py --threshold=10 --max=1 |
| 59 | +# Find problems needing enhancement |
| 60 | +poetry run python .templates/check_test_cases.py --threshold=10 |
158 | 61 |
|
159 | 62 | # Test specific problem |
160 | 63 | make p-test PROBLEM={problem_name} |
161 | 64 |
|
162 | 65 | # Generate from JSON template |
163 | 66 | make p-gen PROBLEM={problem_name} FORCE=1 |
164 | 67 |
|
165 | | -# Lint specific problem |
| 68 | +# Lint check |
166 | 69 | make p-lint PROBLEM={problem_name} |
167 | 70 | ``` |
168 | 71 |
|
169 | | -## Error Handling |
170 | | - |
171 | | -- **Implementation errors**: Ask user before modifying solution code |
172 | | -- **Test failures**: Update JSON template and regenerate |
173 | | -- **Lint failures**: Fix JSON format and iterate |
174 | | -- **Backup failures**: Ensure `.cache/leetcode/` directory exists |
175 | | - |
176 | 72 | ## Success Criteria |
177 | 73 |
|
178 | 74 | - All tests pass with enhanced test cases |
179 | | -- Total test cases exceed the threshold used for detection |
180 | | -- Minimum 10 comprehensive test cases per problem |
181 | | -- Original solution code preserved and working |
| 75 | +- Minimum 12 comprehensive test cases per problem |
| 76 | +- Original solution code preserved |
| 77 | +- **Enhanced test cases in final test_solution.py** |
182 | 78 | - JSON template updated for future regeneration |
183 | | -- Clean final state with no temporary files |
184 | | - |
185 | | -## Automation Benefits |
186 | | - |
187 | | -Using the prebuilt scripts provides several advantages: |
188 | | - |
189 | | -### ⚡ **Speed & Efficiency** |
190 | | - |
191 | | -- **Single command execution** - No need to remember complex multi-step workflows |
192 | | -- **Batch processing** - Enhance multiple problems simultaneously |
193 | | -- **Automatic cleanup** - No manual file management required |
194 | | - |
195 | | -### 🛡️ **Safety & Reliability** |
196 | | - |
197 | | -- **Automatic backups** - Original code always preserved |
198 | | -- **Error handling** - Graceful failure recovery |
199 | | -- **Validation checks** - Syntax and structure verification |
200 | | - |
201 | | -### 📊 **Visibility & Tracking** |
202 | | - |
203 | | -- **Progress reporting** - Real-time status updates |
204 | | -- **Result verification** - Automatic test case counting |
205 | | -- **Success metrics** - Clear pass/fail indicators |
206 | | - |
207 | | -### 🔄 **Consistency** |
208 | | - |
209 | | -- **Standardized process** - Same workflow every time |
210 | | -- **Quality assurance** - Built-in lint and validation checks |
211 | | -- **Reproducible results** - Eliminates human error |
212 | | - |
213 | | -**Recommendation**: Always use automation scripts for test case enhancement unless debugging specific issues requires manual intervention. |
0 commit comments