-
Notifications
You must be signed in to change notification settings - Fork 27
feat: allow users to add constraints and validators to ToolParam
#222
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
979a81e to
343929d
Compare
80d2d98 to
71fdd1d
Compare
343929d to
392169d
Compare
71fdd1d to
92e2cde
Compare
392169d to
f5ebd6d
Compare
92e2cde to
a9d001c
Compare
f5ebd6d to
581a62a
Compare
a9d001c to
d0f2c71
Compare
581a62a to
3214073
Compare
d0f2c71 to
03488ff
Compare
cddc32b to
99b3991
Compare
e2b688c to
7f7c9f3
Compare
99b3991 to
52be67d
Compare
7f7c9f3 to
133cc79
Compare
52be67d to
17e9bbf
Compare
133cc79 to
5cf647b
Compare
17e9bbf to
da184da
Compare
5cf647b to
60bd917
Compare
da184da to
5d9d16a
Compare
60bd917 to
3ea2993
Compare
5d9d16a to
95058fe
Compare
3ea2993 to
507902a
Compare
95058fe to
1cddcec
Compare
c74b8cc to
5d5f1bb
Compare
1cddcec to
97d0650
Compare
3257fef to
088602e
Compare
03ce63e to
2090362
Compare
088602e to
00c5bb6
Compare
2090362 to
57c7ef1
Compare
00c5bb6 to
9d0212c
Compare
57c7ef1 to
d41f806
Compare
3a55e47 to
63bbe0b
Compare
e101741 to
87d5ce7
Compare
63bbe0b to
ed2b8e8
Compare
87d5ce7 to
9e7cb49
Compare
ed2b8e8 to
9fbcb00
Compare
b6cf999 to
492d074
Compare
492d074 to
79f4513
Compare
79f4513 to
afc439c
Compare
afc439c to
3a3a04f
Compare
64adca2 to
f3f1bfc
Compare
YoungVor
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Had a few questions and nits
| if len(query) > MAX_REGEX_LENGTH: | ||
| raise ValidationError(f"Regex too long (>{MAX_REGEX_LENGTH} characters)") | ||
|
|
||
| # Strip inline flags at start like (?i), (?m), combined, to avoid duplication |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it right to simply strip this? Wouldn't that change the behavior of what your matching silently?
| - Balanced (), [], {} | ||
| - Quantifiers {m,n} limited to reasonable bounds | ||
| - Limit number of alternations '|' | ||
| - Disallow backreferences (\\1, \\2, ...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to opus, you can also have 'named' backrefs that you aren't catching in the backreference check, but the exotic construct check might catch it
| raise ValidationError("Unsupported inline regex construct") | ||
|
|
||
| # Heuristic ReDoS guard: forbid nested quantifiers like (.+)+, (.*)+, (?:.+){2,} | ||
| if re.search(r"\((?:[^()]*[+*])[^()]*\)\s*[+*]", query): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TIL about ReDoS :D
Opus mentions overlapping alternation as another form of ReDoS, but its not possible to catch every case. For e.g.: "(a|ab)+"
Is there a timeout for the regex validator?
| if param.allowed_values: | ||
| literal_values = tuple(param.allowed_values) | ||
|
|
||
| def _infer_base_type(p: BoundToolParam): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: _infer_param_base_type, or similar
| annotated_type = TypingAnnotated[base_type, AfterValidator(validate_param)] | ||
| if param.has_default: | ||
| model_fields[param.name] = ( | ||
| Optional[annotated_type], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I vaguely remember there being some reason for this - but 'optional' is a different concept than 'default'.. why not allow an optional param without a default?
| @pytest.mark.parametrize( | ||
| "pattern", | ||
| [ | ||
| r"(.+)+", # nested quantifier |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Opus recommended a few more tests:
@pytest.mark.parametrize(
"pattern,expected_error_substring",
[
# Empty/whitespace
(" ", "empty"),
# Too long
("a" * 300, "too long"),
# Unbalanced brackets
("(", "parentheses"),
(")", "parentheses"),
("[a-", "bracket"),
("]", "bracket"),
("a{1,2", "brace"),
("a}", "brace"),
# Quantifier issues
("a{99999}", "Quantifier"),
("a{5,2}", "upper bound"),
("a{1,2,3}", "syntax"),
# Alternation limit
("a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|aa", "alternation"),
# Backreferences
(r"\1", "backreference"),
(r"\5", "backreference"),
(r"\9", "backreference"),
# Unsupported constructs
(r"(?<=a)b", "construct"),
(r"(?<!a)b", "construct"),
(r"(?=a)", "construct"),
(r"(?!a)", "construct"),
# ReDoS patterns
(r"(.+)+", "Nested"),
(r"(.*)+", "Nested"),
(r".*.*", "greedy"),
(r".+.+", "greedy"),
(r"(.*){3}", "Nested"),
(r"(.+){2,5}", "Nested"),
(r"(?:.*){2}", "Nested"),
(r"(?:.+){3}", "Nested"),
# Invalid regex syntax
("[z-a]", ""), # py_validate_regex catches
("*abc", ""), # py_validate_regex catches
],
)```
| with pytest.raises(PydValidationError, match=re.escape("Input should be greater than 0")): | ||
| Model(city_name="SF", age=0, user_names=["Alice", "Bob"]) | ||
|
|
||
| with pytest.raises(PydValidationError, match=re.escape("Input should be a multiple of 2")): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: good to include test to catch gt 120
f3f1bfc to
f67e849
Compare
…me issues that came up
…the dependencies in ci so the tests will run there
f67e849 to
18dd4cf
Compare

TL;DR
Added parameter constraints and validators to tool parameters to enable better validation of user inputs.
What changed?
NumericConstraintandToolParameterConstraintsmessage types to the protocol buffer definitionsToolParametermessage to support numeric bounds (gt, ge, lt, le), multiple_of, min/max length, and regex pattern validationToolParameterto support custom validatorsParamValidatorprotocolRegexValidatorimplementation with comprehensive regex validationHow to test?
Why make this change?
This change enhances the tool parameter system by adding robust validation capabilities. By supporting both declarative constraints (like min/max values, length limits) and custom validators, we can ensure that user inputs meet specific requirements before executing tool functions. This improves error handling, security, and user experience by providing immediate feedback on invalid inputs rather than failing during execution.