Simplify ray contraint (apache#2573)

# Rationale for this change

We don't support `<3.9`, so lets get rid of complexity

## Are these changes tested?

## Are there any user-facing changes?

<!-- In the case of user-facing changes, please add the changelog label.
-->

Author: Fokko