Skip to content

[Refactor] Remove data loader to simplify api #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
haochengxia merged 4 commits into from
Aug 5, 2025
Merged

[Refactor] Remove data loader to simplify api #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0db2e16
Feat: Make data load internal
haochengxia Aug 5, 2025
ef10981
Update docs for data loading
haochengxia Aug 5, 2025
b187010
Ruff format
haochengxia Aug 5, 2025
5c87c4b
Update examples and remove not used import
haochengxia Aug 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 24 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,33 +54,37 @@ With libcachesim installed, you can start cache simulation for some eviction alg
```python
import libcachesim as lcs

# Step 1: Get one trace from S3 bucket
URI = "cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
dl = lcs.DataLoader()
dl.load(URI)

# Step 2: Open trace and process efficiently
# Step 1: Open a trace hosted on S3 (find more via https://github.com/cacheMon/cache_dataset)
URI = "s3://cache-datasets/cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
reader = lcs.TraceReader(
trace = dl.get_cache_path(URI),
trace = URI,
trace_type = lcs.TraceType.ORACLE_GENERAL_TRACE,
reader_init_params = lcs.ReaderInitParam(ignore_obj_size=False)
)

# Step 3: Initialize cache
cache = lcs.S3FIFO(cache_size=1024*1024)

# Step 4: Process entire trace efficiently (C++ backend)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
# Step 2: Initialize cache
cache = lcs.S3FIFO(
cache_size=1024*1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)

# Step 4.1: Process with limited number of requests
cache = lcs.S3FIFO(cache_size=1024*1024)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(
reader,
start_req=0,
max_req=1000
# Step 3: Process entire trace efficiently (C++ backend)
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")

# Step 3.1: Further process the first 1000 requests again
cache = lcs.S3FIFO(
cache_size=1024 * 1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
```

## Plugin System
Expand Down
44 changes: 24 additions & 20 deletions docs/src/en/getting_started/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,33 +56,37 @@ With libcachesim installed, you can start cache simulation for some eviction alg
```python
import libcachesim as lcs

# Step 1: Get one trace from S3 bucket
URI = "cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
dl = lcs.DataLoader()
dl.load(URI)

# Step 2: Open trace and process efficiently
# Step 1: Open a trace hosted on S3 (find more via https://github.com/cacheMon/cache_dataset)
URI = "s3://cache-datasets/cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
reader = lcs.TraceReader(
trace = dl.get_cache_path(URI),
trace = URI,
trace_type = lcs.TraceType.ORACLE_GENERAL_TRACE,
reader_init_params = lcs.ReaderInitParam(ignore_obj_size=False)
)

# Step 3: Initialize cache
cache = lcs.S3FIFO(cache_size=1024*1024)

# Step 4: Process entire trace efficiently (C++ backend)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
# Step 2: Initialize cache
cache = lcs.S3FIFO(
cache_size=1024*1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)

# Step 4.1: Process with limited number of requests
cache = lcs.S3FIFO(cache_size=1024*1024)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(
reader,
start_req=0,
max_req=1000
# Step 3: Process entire trace efficiently (C++ backend)
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")

# Step 3.1: Further process the first 1000 requests again
cache = lcs.S3FIFO(
cache_size=1024 * 1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
```

The above example demonstrates the basic workflow of using `libcachesim` for cache simulation:
Expand Down
33 changes: 17 additions & 16 deletions examples/basic_usage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,26 @@
import libcachesim as lcs

# Step 1: Get one trace from S3 bucket
URI = "cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
dl = lcs.DataLoader()
dl.load(URI)

# Step 2: Open trace and process efficiently
# Step 1: Open a trace hosted on S3 (find more via https://github.com/cacheMon/cache_dataset)
URI = "s3://cache-datasets/cache_dataset_oracleGeneral/2007_msr/msr_hm_0.oracleGeneral.zst"
reader = lcs.TraceReader(
trace=dl.get_cache_path(URI),
trace=URI,
trace_type=lcs.TraceType.ORACLE_GENERAL_TRACE,
reader_init_params=lcs.ReaderInitParam(ignore_obj_size=False),
)

# Step 3: Initialize cache
cache = lcs.S3FIFO(cache_size=1024 * 1024)
# Step 2: Initialize cache
cache = lcs.S3FIFO(
cache_size=1024 * 1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)

# Step 4: Process entire trace efficiently (C++ backend)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
# Step 3: Process entire trace efficiently (C++ backend)
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")

# Step 4.1: Process with limited number of requests
cache = lcs.S3FIFO(cache_size=1024 * 1024)
obj_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Object miss ratio: {obj_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
# Step 3.1: Further process the first 1000 requests again
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
Comment on lines +24 to +26
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The example is missing the re-initialization of the cache object before the second process_trace call. The process_trace method is stateful and modifies the cache. To correctly process the "first 1000 requests again" from a clean state, the cache should be re-created.

Suggested change
# Step 3.1: Further process the first 1000 requests again
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")
# Step 3.1: Further process the first 1000 requests again
cache = lcs.S3FIFO(
cache_size=1024 * 1024,
# Cache specific parameters
small_size_ratio=0.2,
ghost_size_ratio=0.8,
move_to_main_threshold=2,
)
req_miss_ratio, byte_miss_ratio = cache.process_trace(reader, start_req=0, max_req=1000)
print(f"Request miss ratio: {req_miss_ratio:.4f}, Byte miss ratio: {byte_miss_ratio:.4f}")

3 changes: 0 additions & 3 deletions libcachesim/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@
from .trace_analyzer import TraceAnalyzer
from .synthetic_reader import SyntheticReader, create_zipf_requests, create_uniform_requests
from .util import Util
from .data_loader import DataLoader

__all__ = [
# Core classes
Expand Down Expand Up @@ -118,8 +117,6 @@
"create_uniform_requests",
# Utilities
"Util",
# Data loader
"DataLoader",
# Metadata
"__doc__",
"__version__",
Expand Down
Loading
Loading