Add metrics for sampled policy age in replay buffer #524
+21
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds three new metrics to track the policy age of episodes that are actually sampled from the replay buffer: - buffer/sample/avg_sampled_policy_age: Average age of sampled episodes - buffer/sample/max_sampled_policy_age: Maximum age of sampled episodes - buffer/sample/min_sampled_policy_age: Minimum age of sampled episodes This is distinct from the existing buffer/evict/avg_policy_age metric which tracks the age of all episodes remaining in the buffer after eviction. The new metrics provide visibility into whether training is using fresh data (low ages) or stale data (high ages) at sampling time. Test Plan: - Ran existing unit tests: python -m pytest tests/unit_tests/test_replay_buffer.py -v - All 8 tests passed
meta-cla
bot
added
the
CLA Signed
Contributor
Author
|
cc @wukaixingxp this might help with monitoring your training |
Contributor
|
hey @casteryh , lets wait until you run it to merge? Also, should we then delete the other metric? it seems that maybe its not as useful? |
Contributor
Author
there is no rush merging it. just to make this available so @wukaixingxp can cherry-pick and debug his training jobs.
Yeah it makes sense to me. What you think @joecummings ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
buffer/sample/avg_sampled_policy_age: Average age of sampled episodesbuffer/sample/max_sampled_policy_age: Maximum age of sampled episodesbuffer/sample/min_sampled_policy_age: Minimum age of sampled episodesMotivation
This is distinct from the existing
buffer/evict/avg_policy_agemetric which tracks the age of all episodes remaining in the buffer after eviction. The new metrics provide visibility into whether training is using fresh data (low ages) or stale data (high ages) at sampling time.Test plan
python -m pytest tests/unit_tests/test_replay_buffer.py -vWARNING: Haven't actually run it since I don't want to kill the job that's been running for 2 days on my devgpu