Skip to content

Commit 59241d5

Browse files
committed
fix(dsp): AWS::max_new_token is redundant to LM::max_tokens
Setting max_new_token did not work to increase generated token number for Antropic, but max_tokens does. According to: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html max_tokens is the correct parameter
1 parent 55c91f5 commit 59241d5

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

dsp/modules/aws_models.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,9 @@ def __init__(
4949
self._max_context_size: int = max_context_size
5050
self._max_new_tokens: int = max_new_tokens
5151

52+
# make it consistent with equivalent LM::max_token
53+
self.kwargs["max_tokens"] = max_new_tokens
54+
5255
self.kwargs = {
5356
**self.kwargs,
5457
**kwargs,
@@ -63,7 +66,7 @@ def _call_model(self, body: str) -> str | list[str]:
6366
"""Call model, get generated input without the formatted prompt."""
6467

6568
def _estimate_tokens(self, text: str) -> int:
66-
return len(text)/CHARS2TOKENS
69+
return len(text) / CHARS2TOKENS
6770

6871
def _extract_input_parameters(
6972
self,

0 commit comments

Comments
 (0)