Add batch invariant RL post #113
Open
+86
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Bram Wasti <bwasti@fb.com>
Signed-off-by: Bram Wasti <bwasti@fb.com>
Downloaded GitHub-hosted images to local assets directory and updated all image references to use local paths. Converted standalone images to markdown syntax while keeping centered images as HTML img tags for proper rendering. Signed-off-by: Bram Wasti <bwasti@meta.com>
Added Jekyll frontmatter with layout, title, and author metadata to properly render the blog post. Signed-off-by: Bram Wasti <bwasti@meta.com>
Restored the original width and height attributes (340x130 and 480x355) for the two centered images to maintain their fixed sizing. Signed-off-by: Bram Wasti <bwasti@meta.com>
Added horizontal rule and italic formatting to the acknowledgements section for better visual separation and styling. Signed-off-by: Bram Wasti <bwasti@meta.com>
zhuohan123
reviewed
Nov 10, 2025
WoosukKwon
reviewed
Nov 10, 2025
|
|
||
| In the septillions of flops used to pre-train models, this mismatch between values has largely been avoidable. Pre-training typically runs at a fixed batch size which induces the same reduction kernels to be run - often side-stepping the issue entirely. | ||
|
|
||
| Reinforcement learning, on the other hand, seems to almost exclusively run different reduction algorithms due to its inference-heavy (and thus largely latency and memory-bound) nature. Kernels optimized for low-batch size inference typically run reductions all at once, whereas kernels for training models parallelize heavily to reuse data and amp up compute utilization. That means the generators and the trainers are typically running completely different kernels! |
Collaborator
There was a problem hiding this comment.
Kernels optimized for low-batch size inference typically run reductions all at once
I don't understand this part. Are you talking about reductions like in RMS norm?
Author
There was a problem hiding this comment.
in the kernels, they don't tile. let me use the word "tile" for clarity
youkaichao
reviewed
Nov 11, 2025
youkaichao
reviewed
Nov 11, 2025
youkaichao
reviewed
Nov 11, 2025
Signed-off-by: Bram Wasti <bwasti@fb.com>
Signed-off-by: Bram Wasti <bwasti@fb.com>
Signed-off-by: Bram Wasti <bwasti@fb.com>
Signed-off-by: Bram Wasti <bwasti@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As in title, text can be found in PR content