Skip to content

Commit 8eed623

Browse files
authored
Update MultiPL-E to v3 prompts
I've put together a new MultiPL-E release, v3 and this updates the harness to point to the new prompts. There are fixes to old prompts as well as support for new languages, both described on the MultiPL-E dataset page. The nature of fixes is that some PLs have their scores go up a little. I've tested with with StarCoder2-15B (using VLLM) and will continue testing other models.
1 parent 334efb7 commit 8eed623

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

bigcode_eval/tasks/multiple.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ class GeneralMultiPLE(Task):
8080

8181
DATASET_PATH = "nuprl/MultiPL-E"
8282
DATASET_NAME = None
83-
DATASET_REVISION = "d23b094346c5dbda1080a74bb2a24c18adbf7409"
83+
DATASET_REVISION = "8a4cb75204eb3d5855a81778db6b95bfc80c9136"
8484

8585
def __init__(self, language):
8686
self.language = language

0 commit comments

Comments
 (0)