Skip to content

Conversation

@rjsamra
Copy link
Member

@rjsamra rjsamra commented Apr 12, 2025

Add Streaming Support for Large Excel Files

Description

This PR adds streaming support for processing large Excel files efficiently, preventing memory exhaustion when handling massive datasets.

Features Added

  • New stream() method in ExcelTo class for processing large files
  • Chunk-based processing with configurable chunk size
  • Memory-efficient handling of large datasets
  • Consistent output format with existing methods
  • Comprehensive test coverage

Technical Details

  • Implemented ProcessExcelChunk job for handling data in chunks
  • Added streaming support to ExcelTo class
  • Maintained backward compatibility with existing methods
  • Added feature tests for streaming functionality
  • Updated documentation with streaming feature details

Testing

  • Added comprehensive feature tests
  • Tested with various file sizes and chunk configurations
  • Verified memory usage with large files
  • Tested single and multiple sheet scenarios

Documentation

  • Updated README with streaming feature details
  • Added usage examples
  • Documented response formats
  • Added memory efficiency notes

Breaking Changes

None. This is a backward-compatible feature addition.

Checklist

  • Code follows the project's coding standards
  • Tests have been added/updated
  • Documentation has been updated
  • All tests pass
  • No breaking changes introduced

@rjsamra rjsamra self-assigned this Apr 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants