Skip to content

Commit 8fc2d76

Browse files
committed
docs: add test-case-enhancement.md
1 parent 24eeece commit 8fc2d76

File tree

2 files changed

+182
-12
lines changed

2 files changed

+182
-12
lines changed
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Test Case Enhancement Rules
2+
3+
## Assistant Workflow for Adding Comprehensive Test Cases
4+
5+
When user requests to enhance test cases for a problem, the assistant will:
6+
7+
### 1. Problem Resolution (Priority Order)
8+
9+
- **FIRST**: Try to resolve from context - check active file path or user-provided problem name
10+
- **SECOND**: If context resolution fails, THEN run `poetry run python .templates/check_test_cases.py --threshold=10 --max=1` to auto-detect 1 problem with <10 test cases
11+
- **LAST**: If both above fail, ask user to explicitly specify problem name
12+
13+
### 2. Test Case Generation
14+
15+
- Read `leetcode/{problem_name}/README.md` for problem understanding
16+
- Analyze existing test cases in `leetcode/{problem_name}/tests.py`
17+
- Generate comprehensive test cases covering:
18+
- **Edge cases**: Empty inputs, single elements, boundary values
19+
- **Corner cases**: Maximum/minimum constraints, special patterns
20+
- **Normal cases**: Typical scenarios with varied complexity
21+
- **Error cases**: Invalid inputs (if applicable)
22+
23+
### 3. Initial Validation
24+
25+
- Run `make p-test PROBLEM={problem_name}` to verify current implementation
26+
- **If errors found**:
27+
- DO NOT update implementation automatically
28+
- Only update test cases if they're incorrect
29+
- If implementation seems wrong, ASK USER first before modifying
30+
31+
### 4. JSON Template Update
32+
33+
- Update corresponding `.templates/leetcode/json/{problem_name}.json`
34+
- Add new test cases to `test_cases` field in proper format
35+
- Maintain existing test structure and naming conventions
36+
37+
### 5. Backup and Regeneration Process
38+
39+
- **Backup**: Move `leetcode/{problem_name}/` to `.cache/leetcode/{problem_name}/`
40+
- **Regenerate**: Run `make p-gen PROBLEM={problem_name} FORCE=1`
41+
- **Lint check**: Run `make p-lint PROBLEM={problem_name}`
42+
- **Iterate**: If lint fails, update JSON and regenerate until passes
43+
44+
### 6. Solution Preservation
45+
46+
- Copy `solution.py` from backup to newly generated structure
47+
- Run `make p-test PROBLEM={problem_name}` to verify tests pass
48+
- **If tests fail**: Go back to step 4, update JSON, and iterate until passes
49+
50+
### 7. Cleanup and Restore
51+
52+
- Remove newly generated files
53+
- Restore original structure from backup
54+
- Verify final state with `make p-test PROBLEM={problem_name}`
55+
56+
## Test Case Quality Standards
57+
58+
### Coverage Requirements
59+
60+
- **Minimum 10 test cases** per problem
61+
- **Edge cases**: 20-30% of total test cases
62+
- **Normal cases**: 50-60% of total test cases
63+
- **Corner cases**: 20-30% of total test cases
64+
65+
### Test Case Categories
66+
67+
#### Edge Cases
68+
69+
- Empty inputs: `[]`, `""`, `None`
70+
- Single element: `[1]`, `"a"`
71+
- Boundary values: `[0]`, `[1]`, `[-1]`
72+
- Maximum/minimum constraints from problem description
73+
74+
#### Corner Cases
75+
76+
- Duplicate elements: `[1,1,1]`
77+
- Sorted/reverse sorted arrays: `[1,2,3]`, `[3,2,1]`
78+
- All same elements: `[5,5,5,5]`
79+
- Alternating patterns: `[1,0,1,0]`
80+
81+
#### Normal Cases
82+
83+
- Mixed positive/negative numbers
84+
- Various array sizes within constraints
85+
- Different data patterns and structures
86+
- Representative problem scenarios
87+
88+
### JSON Format Requirements
89+
90+
- Use single quotes for Python strings in test cases
91+
- Follow existing parametrize format
92+
- Maintain type hints in parametrize_typed
93+
- Ensure test_cases string is valid Python list syntax
94+
95+
## Commands Reference
96+
97+
```bash
98+
# Find problems needing more test cases
99+
poetry run python .templates/check_test_cases.py --threshold=10 --max=1
100+
101+
# Test specific problem
102+
make p-test PROBLEM={problem_name}
103+
104+
# Generate from JSON template
105+
make p-gen PROBLEM={problem_name} FORCE=1
106+
107+
# Lint specific problem
108+
make p-lint PROBLEM={problem_name}
109+
```
110+
111+
## Error Handling
112+
113+
- **Implementation errors**: Ask user before modifying solution code
114+
- **Test failures**: Update JSON template and regenerate
115+
- **Lint failures**: Fix JSON format and iterate
116+
- **Backup failures**: Ensure `.cache/leetcode/` directory exists
117+
118+
## Success Criteria
119+
120+
- All tests pass with enhanced test cases
121+
- Minimum 10 comprehensive test cases per problem
122+
- Original solution code preserved and working
123+
- JSON template updated for future regeneration
124+
- Clean final state with no temporary files

.templates/check_test_cases.py

Lines changed: 58 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22

33
import json
44
from pathlib import Path
5+
from typing import Optional
6+
import typer
7+
58

69
def count_test_cases(json_data):
710
"""Count total test cases across all test methods."""
@@ -15,31 +18,74 @@ def count_test_cases(json_data):
1518
for method in test_methods:
1619
test_cases = method.get("test_cases", "")
1720
if test_cases.strip():
18-
# Count tuples/lists in test_cases string
19-
total += max(test_cases.count("("), test_cases.count("["))
21+
# Parse the test_cases string to count actual test cases
22+
try:
23+
# Remove outer brackets and split by top-level commas
24+
cases_str = test_cases.strip()
25+
if cases_str.startswith("[") and cases_str.endswith("]"):
26+
cases_str = cases_str[1:-1] # Remove outer brackets
27+
28+
# Count test cases by counting commas at parenthesis depth 0
29+
depth = 0
30+
case_count = 1 if cases_str.strip() else 0
31+
32+
for char in cases_str:
33+
if char in "([{":
34+
depth += 1
35+
elif char in ")]}":
36+
depth -= 1
37+
elif char == "," and depth == 0:
38+
case_count += 1
39+
40+
total += case_count
41+
except Exception:
42+
# Fallback to old method if parsing fails
43+
total += test_cases.count("(") - test_cases.count("([") + test_cases.count("[(")
2044
return total
2145

22-
def main():
46+
47+
def main(
48+
threshold: int = typer.Option(
49+
10, "--threshold", "-t", help="Show files with test cases <= threshold"
50+
),
51+
max_results: str = typer.Option(
52+
1, "--max", "-m", help="Maximum number of results to show ('none' for no limit)"
53+
),
54+
):
55+
"""Check test case counts in LeetCode JSON templates."""
2356
json_dir = Path(".templates/leetcode/json")
24-
files_with_few_tests = []
57+
all_files = []
2558

2659
for json_file in json_dir.glob("*.json"):
2760
try:
2861
with open(json_file) as f:
2962
data = json.load(f)
3063

3164
test_count = count_test_cases(data)
32-
if test_count <= 10:
33-
files_with_few_tests.append((json_file.name, test_count))
65+
all_files.append((json_file.name, test_count))
3466
except Exception as e:
35-
print(f"Error reading {json_file.name}: {e}")
67+
typer.echo(f"Error reading {json_file.name}: {e}", err=True)
3668

3769
# Sort by test count
38-
files_with_few_tests.sort(key=lambda x: x[1])
70+
all_files.sort(key=lambda x: x[1])
71+
72+
# Filter by threshold
73+
filtered_files = [f for f in all_files if f[1] <= threshold]
74+
75+
# Apply max results limit
76+
if max_results.lower() not in ["none", "null", "-1"]:
77+
try:
78+
max_count = int(max_results)
79+
if max_count > 0:
80+
filtered_files = filtered_files[:max_count]
81+
except ValueError:
82+
typer.echo(f"Invalid max_results value: {max_results}", err=True)
83+
raise typer.Exit(1)
84+
85+
typer.echo(f"Files with ≤{threshold} test cases ({len(filtered_files)} total):")
86+
for filename, count in filtered_files:
87+
typer.echo(f"{filename}: {count} test cases")
3988

40-
print(f"Files with ≤10 test cases ({len(files_with_few_tests)} total):")
41-
for filename, count in files_with_few_tests:
42-
print(f"{filename}: {count} test cases")
4389

4490
if __name__ == "__main__":
45-
main()
91+
typer.run(main)

0 commit comments

Comments
 (0)