Skip to content

fix: Add batching for visitors data loading for Matomo to avoid too-l… #666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

fix: Add batching for visitors data loading for Matomo to avoid too-l… #666

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7e84ba8
fix: Add batching for visitors data loading for Matomo to avoid too-l…
NHNguyen94 Oct 29, 2025
19f6321
fix: Change default visitors loading function to the batching one
NHNguyen94 Oct 29, 2025
600c9fe
chore: Remove redundant func for get unique visitors without chunk
NHNguyen94 Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions sources/matomo/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,23 +197,28 @@ def get_last_visits(
primary_key="visitorId",
)
def get_unique_visitors(
visits: List[DictStrAny], client: MatomoAPIClient, site_id: int
visits: List[DictStrAny],
client: MatomoAPIClient,
site_id: int,
chunk_size: int = 20,
) -> Iterator[TDataItem]:
"""
Dlt transformer. Receives information about visits from get_last_visits.
This version allows batch loading for visitors data, which is to avoid too-long-URL issue

Args:
visits (List[DictStrAny]): List of dicts containing information on last visits in the given timeframe.
client (MatomoAPIClient): Used to make calls to Matomo API.
site_id (int): Every site in Matomo has a unique id.
chunk_size (int): Number of visitor IDs to process in each batch. Defaults to 20.

Returns:
Iterator[TDataItem]: Dict containing information about the visitor.
"""

visitor_ids = [visit["visitorId"] for visit in visits]
indexed_visitor_ids = [
visitor_ids[i : i + 100] for i in range(0, len(visitor_ids), 100)
visitor_ids[i : i + chunk_size] for i in range(0, len(visitor_ids), chunk_size)
]
for visitor_list in indexed_visitor_ids:
method_data = client.get_visitors_batch(
Expand Down