1410 Accelerate add_edge for large graphs #1411

kilianvolmer · 2025-11-06T14:23:39Z

Changes and Information

Please briefly list the changes (main added features, changed items, or corrected bugs) made:

Added a lazy_add_edge function that does neither check for uniqueness nor sort the added edges.
Added a sort_edges and a make_edges_unique function
Added a reserve_edges function to reserve space for edges

If need be, add additional information and what the reviewer should look out for in particular:

Merge Request - Guideline Checklist

Please check our git workflow. Use the draft feature if the Pull Request is not yet ready to review.

Checks by code author

Every addressed issue is linked (use the "Closes #ISSUE" keyword below)
New code adheres to coding guidelines
No large data files have been added (files should in sum not exceed 100 KB, avoid PDFs, Word docs, etc.)
Tests are added for new functionality and a local test run was successful (with and without OpenMP)
Appropriate documentation within the code (Doxygen) for new functionality has been added in the code
Appropriate external documentation (ReadTheDocs) for new functionality has been added to the online documentation
Proper attention to licenses, especially no new third-party software with conflicting license has been added
(For ABM development) Checked benchmark results and ran and posted a local test above from before and after development to ensure performance is monitored.

Checks by code reviewer(s)

Corresponding issue(s) is/are linked and addressed
Code is clean of development artifacts (no deactivated or commented code lines, no debugging printouts, etc.)
Appropriate unit tests have been added, CI passes, code coverage and performance is acceptable (did not decrease)
No large data files added in the whole history of commits(files should in sum not exceed 100 KB, avoid PDFs, Word docs, etc.)
On merge, add 2-5 lines with the changes (main added features, changed items, or corrected bugs) to the merge-commit-message. This can be taken from the briefly-list-the-changes above (best case) or the separate commit messages (worst case).

Closes #1410

codecov · 2025-11-06T15:28:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.28%. Comparing base (6233316) to head (d14011b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1411      +/-   ##
==========================================
- Coverage   97.29%   97.28%   -0.01%     
==========================================
  Files         180      180              
  Lines       15646    15663      +17     
==========================================
+ Hits        15223    15238      +15     
- Misses        423      425       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

HenrZu

Thanks for the update! Im still not entirely sure about the overall usefulness of the feature.
Do you have some information about the overhead caused by the sorting and checking for dublicates?
if we want to add this feature, we should also extend the documentation of the current add edge function starting in line 169.

HenrZu · 2025-11-12T09:45:49Z

cpp/memilio/mobility/graph.h

+    * @return Edge<EdgePropertyT>& End of edge vector
+    */
+    template <class... Args>
+    Edge<EdgePropertyT>& lazy_add_edge(size_t start_node_idx, size_t end_node_idx, Args&&... args)


we should catch the case, when m_edges is empty

Why? Doesn't emplace_back also work on an empty vector?

He potentially means uninitialized, not empty?

HenrZu · 2025-11-12T09:47:41Z

cpp/memilio/mobility/graph.h

+     * 
+     * @return Edge<EdgePropertyT>& End of edge vector
+     */
+    Edge<EdgePropertyT>& sort_edges()


why return the reference and not use void?

HenrZu · 2025-11-12T09:51:29Z

cpp/memilio/mobility/graph.h

+    /**
+     * @brief Make the edges of a graph unique.
+     * 
+     * Copies all the unique edges to a new vector and replaces the edge vector of the graph with it. Unique means that 


also mention that properties might get lost.
Additionally, if i add two edges with the same start/end combination. Which edges gets deleted? the one that was added latest?

HenrZu · 2025-11-12T09:53:56Z

cpp/memilio/mobility/graph.h

+     * the start and end node indices are unique. Other edge properties are not checked.
+     * @return Edge<EdgePropertyT>& End of edge vector
+     */
+    Edge<EdgePropertyT>& make_edges_unique()


i think it would be clearer, if we use the term "remove dublicates" instead of unique edges.
Also renaming the function to remove_duplicate_edges would add clarity here imo.

HenrZu · 2025-11-12T09:55:56Z

cpp/memilio/mobility/graph.h

+     * @return Edge<EdgePropertyT>& End of edge vector
+     */
+    Edge<EdgePropertyT>& make_edges_unique()
+    {


maybe also an option to merge the properties would be nice. However, this would make some restrictions to the property.

HenrZu · 2025-11-12T10:06:06Z

cpp/memilio/mobility/graph.h

+    {
+        std::vector<Edge<EdgePropertyT>> unique_edges;
+        unique_edges.reserve(m_edges.size());
+        std::ranges::unique_copy(m_edges, std::back_inserter(unique_edges), [](auto&& e1, auto&& e2) {


we do a copy here. Is this really neccessary?

HenrZu · 2025-11-12T10:10:40Z

cpp/memilio/mobility/graph.h

+    * @param start_node_idx Id of start node
+    * @param end_node_idx Id of end node
+    * @param args Additional arguments for edge construction
+    * @return Edge<EdgePropertyT>& End of edge vector


describe the return more in detail. This is to generic.
Currently, its a reference to the latest added edge.

Could you also doxy-comment @tparam ?

HenrZu · 2025-11-12T10:11:56Z

cpp/memilio/mobility/graph.h

+    * @return Edge<EdgePropertyT>& End of edge vector
+    */
+    template <class... Args>
+    Edge<EdgePropertyT>& lazy_add_edge(size_t start_node_idx, size_t end_node_idx, Args&&... args)


Can you also add a motivation, why we even need this function? Whats the use case? why not add a bool flag to the existing add_edge function to allow addition without checking or sorting?

also, i would prefer the name add_edge_naive

or maybe directly unchecked?

HenrZu · 2025-11-12T10:14:03Z

cpp/tests/test_graph.cpp

    EXPECT_THAT(g.out_edges(1), testing::ElementsAreArray(v1));
 }

+TEST(TestGraph, compare_add_edge_functions)


Test cases are great so far. However, could you add a few more tests for edge cases (empty graph / calling make_unique... before sorting / testing if the correct properties remain and the expected ones get deleted)

HenrZu · 2025-11-12T10:19:04Z

docs/source/cpp/graph_metapop.rst


+.. dropdown:: :fa:`gears` Working with large graphs
+
+    When working with very large graphs, i.e. starting from a few thousand edges, it will be faster to not use the standard ``add_edge`` function. This function always


Ok, now i see the motivation. Some questions.
Does that really make a noticeable difference? Have you ever quantified it?

Especially in simulations with many edges, we expect a relatively large runtime. Is the overhead caused by checking so much higher that we accept new sources of error (such as forgetting to call the sort function, etc.)?

Yes, it does make a difference of around four orders of magnitude in example simulations. For example, with 1.6 million edges I measure 4.5e+03 seconds with the standard function and 9.6e-01 seconds with the new functions.

CHG: Add lazy edge addition function

42accae

kilianvolmer requested a review from HenrZu November 6, 2025 14:23

kilianvolmer changed the title ~~1410-Accelerate-add_edge-for-large-graphs~~ 1410-Accelerate add_edge for large graphs Nov 6, 2025

kilianvolmer changed the title ~~1410-Accelerate add_edge for large graphs~~ 1410 Accelerate add_edge for large graphs Nov 6, 2025

FIX: start edge generation with correct node index

d14011b

HenrZu requested changes Nov 12, 2025

View reviewed changes


		.. dropdown:: :fa:`gears` Working with large graphs

		When working with very large graphs, i.e. starting from a few thousand edges, it will be faster to not use the standard ``add_edge`` function. This function always

1410 Accelerate add_edge for large graphs #1411

Are you sure you want to change the base?

1410 Accelerate add_edge for large graphs #1411

Uh oh!

Conversation

kilianvolmer commented Nov 6, 2025 • edited by HenrZu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes and Information

Merge Request - Guideline Checklist

Checks by code author

Checks by code reviewer(s)

Uh oh!

codecov bot commented Nov 6, 2025

Codecov Report

Uh oh!

HenrZu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilianvolmer Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilianvolmer Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kilianvolmer commented Nov 6, 2025 •

edited by HenrZu

Loading

kilianvolmer Nov 12, 2025 •

edited

Loading

kilianvolmer Nov 12, 2025 •

edited

Loading