⚡️ Speed up function compare_test_results by 655% in PR #945 (feat/feedback-loop-for-unmatched-test-results)
#947
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The optimizations deliver a **655% speedup** by addressing several key bottlenecks identified in the line profiler results: **Key Optimizations:** 1. **Caching parsed CST modules** - Added `@lru_cache` to `InvocationId._parse_module_by_path()` to avoid repeatedly parsing the same test files. The original code spent 68% of time in `cst.parse_module()`, which is now cached for repeated calls. 2. **Single-pass AST traversal** - Combined class and function search into one loop with early returns, eliminating redundant iterations through `module_node.body`. 3. **Optimized dictionary lookups** - In `TestResults.get_by_unique_invocation_loop_id()`, replaced the try/except pattern with direct `.get()` calls to avoid exception overhead. 4. **Reordered type checks in comparator** - Moved cheap, common types (str, int, bool) to the front of isinstance checks, allowing ~75% of comparisons to exit early without checking expensive types like numpy arrays. 5. **Eliminated generator allocation** - Replaced `all()` comprehensions with direct for-loops that can break early, avoiding unnecessary iteration over remaining elements. 6. **Cached function references** - In the hot loop of `compare_test_results()`, cached method lookups like `get_by_unique_invocation_loop_id` to avoid repeated attribute resolution. **Impact on Hot Paths:** Based on the function references, this code is called in the critical path of `run_optimized_candidate()`, which executes during performance testing of optimization candidates. The speedup means: - Faster validation of test result equivalence between original and optimized code - Reduced overhead when processing many test results with repeated file parsing - More efficient comparison of complex data structures in test outputs The optimizations are particularly effective for workloads with many test invocations on the same files and complex return value comparisons, which matches the typical usage pattern shown in the function references.
codeflash-ai
bot
added
⚡️ codeflash
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
⚡️ This pull request contains optimizations for PR #945
If you approve this dependent PR, these changes will be merged into the original PR branch
feat/feedback-loop-for-unmatched-test-results.📄 655% (6.55x) speedup for
compare_test_resultsincodeflash/verification/equivalence.py⏱️ Runtime :
90.0 milliseconds→11.9 milliseconds(best of5runs)📝 Explanation and details
The optimizations deliver a 655% speedup by addressing several key bottlenecks identified in the line profiler results:
Key Optimizations:
Caching parsed CST modules - Added
@lru_cachetoInvocationId._parse_module_by_path()to avoid repeatedly parsing the same test files. The original code spent 68% of time incst.parse_module(), which is now cached for repeated calls.Single-pass AST traversal - Combined class and function search into one loop with early returns, eliminating redundant iterations through
module_node.body.Optimized dictionary lookups - In
TestResults.get_by_unique_invocation_loop_id(), replaced the try/except pattern with direct.get()calls to avoid exception overhead.Reordered type checks in comparator - Moved cheap, common types (str, int, bool) to the front of isinstance checks, allowing ~75% of comparisons to exit early without checking expensive types like numpy arrays.
Eliminated generator allocation - Replaced
all()comprehensions with direct for-loops that can break early, avoiding unnecessary iteration over remaining elements.Cached function references - In the hot loop of
compare_test_results(), cached method lookups likeget_by_unique_invocation_loop_idto avoid repeated attribute resolution.Impact on Hot Paths:
Based on the function references, this code is called in the critical path of
run_optimized_candidate(), which executes during performance testing of optimization candidates. The speedup means:The optimizations are particularly effective for workloads with many test invocations on the same files and complex return value comparisons, which matches the typical usage pattern shown in the function references.
✅ Correctness verification report:
⚙️ Existing Unit Tests and Runtime
test_codeflash_capture.py::test_codeflash_capture_basictest_codeflash_capture.py::test_codeflash_capture_multiple_helperstest_codeflash_capture.py::test_codeflash_capture_recursivetest_codeflash_capture.py::test_codeflash_capture_super_inittest_codeflash_capture.py::test_instrument_codeflash_capture_and_run_teststest_codeflash_capture.py::test_instrument_codeflash_capture_and_run_tests_2test_comparator.py::test_compare_results_fntest_instrument_all_and_run.py::test_bubble_sort_behavior_resultstest_instrument_all_and_run.py::test_classmethod_full_instrumentationtest_instrument_all_and_run.py::test_method_full_instrumentationtest_instrument_all_and_run.py::test_staticmethod_full_instrumentationtest_instrumentation_run_results_aiservice.py::test_class_method_full_instrumentationtest_instrumentation_run_results_aiservice.py::test_class_method_test_instrumentation_onlytest_pickle_patcher.py::test_run_and_parse_picklepatchTo edit these changes
git checkout codeflash/optimize-pr945-2025-11-27T18.26.50and push.