fix: update the legacy behaviour

Author: terryyz

README.md

@@ -347,7 +347,7 @@ We share pre-generated code samples from LLMs we have [evaluated](https://huggin
 
 ## 🐞 Known Issues
 
-- [ ] Due to [the Hugging Face tokenizer update](https://github.com/huggingface/transformers/pull/31305), some tokenizer may be broken and will degrade the performance of the evaluation. Please try `--tokenizer_legacy` during the generation.
+- [ ] Due to [the Hugging Face tokenizer update](https://github.com/huggingface/transformers/pull/31305), some tokenizer may be broken and will degrade the performance of the evaluation. Therefore, we set up with `legacy=False` for the initialization. If you notice the unexpected change, please try `--tokenizer_legacy` during the generation.
 
 - [ ] Due to the flakes in the evaluation, the execution results may vary slightly (~0.2%) between runs. We are working on improving the evaluation stability.
 

bigcodebench/model.py

@@ -135,7 +135,7 @@ def __init__(self, name: str, dataset: str, tp: int, **kwargs) -> None:
         if self.tokenizer_name is None:
             self.tokenizer_name = self.name
 
-        self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name, **kwargs, legacy=not self.tokenizer_legacy)
+        self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name, **kwargs, legacy=self.tokenizer_legacy)
         if self.tokenizer.chat_template is None:
             self.eos += extra_eos_for_direct_completion(dataset)
         self.llm = LLM(model=name, max_model_len=2048, **kwargs)
@@ -195,7 +195,7 @@ def __init__(self, name: str, dataset: str, **kwargs):
         if self.tokenizer_name is None:
             self.tokenizer_name = self.name
 
-        self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name, **kwargs, legacy=not self.tokenizer_legacy)
+        self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name, **kwargs, legacy=self.tokenizer_legacy)
 
         if self.tokenizer.chat_template is None:
             self.eos += extra_eos_for_direct_completion(dataset)
@@ -252,7 +252,7 @@ def __init__(self, name: str, **kwargs):
         self.eos += ["\n```\n"]
         print(f"EOS strings: {self.eos}")
         self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name if self.tokenizer_name else self.name,
-                                                       **kwargs, legacy=not self.tokenizer_legacy)
+                                                       **kwargs, legacy=self.tokenizer_legacy)
 
     def codegen(
         self, prompt: str, do_sample: bool = True, num_samples: int = 200