perf(serverless): lazy-load boto3, fastapi, and pydantic to reduce cold start time #466
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Move boto3 imports from module level to function level in rp_upload.py. This defers loading ~50MB of boto3/botocore dependencies until S3 upload functions are actually called, improving initial import time and memory footprint for users who don't use S3 features. Changes: - Refactored get_boto_client() and bucket_upload() with lazy imports - Added ImportError handling with helpful error messages - Updated tests to mock boto3 modules directly - Enhanced documentation to explain lazy-loading behavior All upload functions maintain backward compatibility and graceful fallback to local file storage when boto3 is unavailable.
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes serverless cold start performance by deferring boto3 imports until S3 upload functions are actually invoked, reducing initial module load time and memory usage for applications that don't utilize S3 features.
Key Changes:
- Moved boto3/botocore imports from module-level to function-level with try/except handling
- Updated return type annotations from specific boto3 types to generic
Anytypes - Added graceful degradation with local file storage fallback when boto3 is unavailable
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| runpod/serverless/utils/rp_upload.py | Refactored get_boto_client(), bucket_upload(), and upload_in_memory_object() to use lazy-loaded boto3 imports with error handling |
| tests/test_serverless/test_utils/test_upload.py | Updated mock paths from module-level to boto3-level to reflect lazy-loading changes |
| docs/serverless/utils/rp_upload.md | Added Requirements section documenting boto3 dependency and lazy-loading behavior |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review comments from PR #466: 1. Extract boto3 import logic into shared helper function - Created _import_boto3_dependencies() to reduce duplication - Used by both get_boto_client() and bucket_upload() - Consistent error handling across all S3 upload functions 2. Add TransferConfig import to bucket_upload() - Now imports all boto3 dependencies via shared helper - Maintains consistency with get_boto_client() 3. Clarify documentation about fallback directories - Documented that upload_image() uses simulated_uploaded/ - Documented that public API functions use local_upload/ - Added context about when fallback behavior occurs All 358 tests pass with 97% coverage.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…paths Address Copilot review comment from PR #466: Replace print() statements with logger.warning() for consistency with the module's logging setup. This allows proper log level control and maintains consistent logging behavior throughout the module. Changes: - upload_image(): Use logger.warning() instead of print() - upload_file_to_bucket(): Use logger.warning() instead of print() - upload_in_memory_object(): Use logger.warning() instead of print() All fallback messages now use structured logging with single-line format for better log parsing and filtering.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review comment from PR #466: Extract duplicated fallback logic from upload_file_to_bucket() and upload_in_memory_object() into a shared _save_to_local_fallback() helper function to reduce code duplication and ensure consistent behavior. Changes: - Created _save_to_local_fallback() helper function - Handles both file-based (source_path) and in-memory (file_data) uploads - Consolidated logging, directory creation, and file saving logic - upload_file_to_bucket() now calls helper with source_path parameter - upload_in_memory_object() now calls helper with file_data parameter Test improvements: - Added test_upload_file_to_bucket_fallback() for file-based fallback - Added test_upload_in_memory_object_fallback() for in-memory fallback - Added test_save_to_local_fallback_invalid_args() for error handling - Added test_import_boto3_dependencies_missing() for ImportError path - Achieved 100% test coverage for rp_upload.py module Benefits: - Reduced code duplication (removed 12 lines of duplicate code) - Single source of truth for fallback behavior - Easier to maintain and test - Consistent error messages and logging - Complete test coverage ensures reliability All 10 upload tests pass with 100% module coverage.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…nctions Address latest Copilot review comment from PR #466: Eliminate remaining duplication by making upload_image() use the _save_to_local_fallback() helper function. Added a 'directory' parameter to the helper to support different fallback directories. Changes: - Added 'directory' parameter to _save_to_local_fallback() (default: 'local_upload') - Updated upload_image() to use helper with directory='simulated_uploaded' - Removed duplicate warning message and URL from upload_image() - Consolidated all fallback logic into single helper function Benefits: - Complete elimination of code duplication - Single source of truth for all fallback behavior - Consistent warning messages across all upload functions - Easier to maintain and update fallback logic All 362 tests pass with 97% overall coverage, 100% coverage on rp_upload.py.
jhcipar
reviewed
Nov 10, 2025
Use TYPE_CHECKING to import boto3 types only during static type checking, maintaining proper type safety without runtime import cost. Changes: - Import BaseClient and TransferConfig under TYPE_CHECKING guard - Restore get_boto_client() return type from Tuple[Any, Any] to Tuple[Optional[BaseClient], Optional[TransferConfig]] - Remove # pragma: no cover comment as it's no longer needed This addresses PR review feedback about maintaining type safety while preserving the lazy-loading optimization.
Move FastAPI/Uvicorn/Pydantic imports from module-level to conditional blocks where they're actually used. This stack is only needed when --rp_serve_api flag is set (local dev) or realtime mode is enabled. Performance Impact: - Cold start: 480ms → 280-326ms (32-42% faster) - Modules loaded: 841 → 640 (24% reduction, ~200 fewer) - Production workers: Never load FastAPI/Uvicorn/Pydantic stack - Dev mode: FastAPI loads on-demand when needed Changes: - Remove eager import of rp_fastapi from module level - Add lazy import in start() when rp_serve_api flag is True - Add lazy import in start() when realtime mode is enabled All tests pass. No breaking changes.
Update test mocks to use correct import path for lazy-loaded rp_fastapi module. Since FastAPI is now imported on-demand inside start() function rather than at module level, tests need to mock the actual module path. Changes: - Update test_local_api to mock runpod.serverless.modules.rp_fastapi.WorkerAPI - Update test_start_does_not_set_excepthook to mock correct module path All 362 tests pass with 96.76% coverage.
deanq
changed the title
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
runpod/serverless/utils/rp_upload.py:1
- Corrected f-string formatting: removed unnecessary f-prefix from non-interpolated string.
""" PodWorker | modules | upload.py """
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jhcipar
approved these changes
Nov 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement lazy-loading for heavy dependencies in serverless modules to dramatically reduce cold start time. This optimization defers loading boto3 (~50MB) and FastAPI/Uvicorn/Pydantic stack (~300ms) until they're actually needed.
Performance Impact
Serverless Worker Cold Start
Production Impact
Changes
1. boto3 Lazy-Loading (rp_upload.py)
_import_boto3_dependencies()helper functionTYPE_CHECKINGto restore proper type hints without runtime import costTuple[Optional["BaseClient"], Optional["TransferConfig"]]_save_to_local_fallback()helper2. FastAPI Lazy-Loading (serverless/init.py)
rp_fastapifrom module levelstart()function--rp_serve_apiflag is set or realtime mode is enabled3. Error Handling & Fallbacks
print()withlogger.warning()in fallback paths4. Code Quality Improvements
Testing
Dependencies Optimized
Backward Compatibility
All upload functions maintain full backward compatibility with graceful fallback to local file storage when dependencies are unavailable. No changes required to existing code using these functions.