Skip to content

Commit 40dfce6

Browse files
authored
Skip {% generation %} and {% endgeneration %} template handling (#3204)
* Add `.DS_Store` file to `.gitignore` * Skip `{% generation %}` and `{% endgeneration %}` Custom syntax within the chat template for the Phi4 Reasoning models e.g. https://huggingface.co/microsoft/Phi-4-reasoning-plus, which is AFAIK not handled natively yet, so skipping for now * Update explanation on `{% generation %}` and `{% endgeneration %}` removal * Revert "Add `.DS_Store` file to `.gitignore`" This reverts commit d64d6d2.
1 parent e7329fe commit 40dfce6

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

router/src/infer/chat_template.rs

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@ impl ChatTemplate {
4545
// It uses python notation to reverse lists, which do not exist in minijinja
4646
// so we're using the reverse filter instead.
4747
let mutated_template = mutated_template.replace("[::-1]", "|reverse");
48+
// TODO: replace with a better solution
49+
// Hack to remove the {% generation %} and {% endgeneration %} statements from
50+
// the Jinja2 chat templates if there, since those are only using for assistant
51+
// masking during training, and should be ignored during inference
52+
let mutated_template = mutated_template.replace("{% generation %}", "");
53+
let mutated_template = mutated_template.replace("{% endgeneration %}", "");
4854

4955
let template_str = mutated_template.into_boxed_str();
5056
env.add_function("raise_exception", raise_exception);

0 commit comments

Comments
 (0)