Work

EleutherAI/lm-evaluation-harness: v0.4.13

Year: 2021
Type: software
Abstract: v0.4.13 Release Notes A fix-focused release. The main fixes are for few-shot leakage, a multiple-choice filter bug, and group stderr, alongside two new ONNX backends and eight new benchmark suites. Also updated most configs for datasets>=4, which accounts for much of the diff by volume. Highlights Bug Fixes Fixes that may shift previously reported numbers: Eval documents leaked into few-shot prompts. The sampler could draw the document under test into its own demonstrations by @SiavashShams in #3978, and gen_prefix was resolved against the eval doc rather than the few-shot doc — splicing the evaluated question into every shot for RULER niah_single_1, humaneval_instruct, and humaneval_64_instruct by @adityasingh2400 in #3979 MultiChoiceRegexFilter prefix-shadowing. Regex alternation is leftmost-wins, so a choice that prefixed a longer choice ("Guilty" vs "Guilty of Romance") matched inside it and scored correct answers as wrong — visible on BBH movie_recommendation by @iamsharduld in #3884 Group stderr with weight_by_size: false. Groups reported the size-weighted pooled stderr even when the point estimate was an unweighted mean, giving error bars up to ~3x too narrow. Only unequal-sized subtasks change by @iamsharduld in #3882 minerva_math answer normalization. sqrt shorthand no longer corrupts indexed roots (#4037), the thousands-separator strip no longer fuses digit tuples (0,1 → 01, #4039), an answer identical to the gold now scores correct (#4034), and the few-shot prompt LaTeX is corrected (#4045) by @feiiiiii5 and @nata2627. putnam_axiom shares these helpers and picks up the same fixes. New Model Backends onnxruntime — raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in #3984 onnxruntime-genai — cross-platform ONNX Runtime GenAI backend, with winml refactored on top of it by @thiagocrepaldi in #3960 Megatron-LM v0.18 compatibility by @oberpierre in #3991 Install with pip install lm_eval[onnxruntime] or lm_eval[onnxruntime-genai]. For non-CPU execution providers install the matching wheel instead — onnxruntime-gpu / onnxruntime-rocm, or onnxruntime-genai-cuda / onnxruntime-genai-directml — these are mutually exclusive. New Tasks LongProc — long-context procedural reasoning, 6 task types across 16 configs by @xiye17 in #3544 Uncheatable Eval — contamination-resistant evaluation: rolling log-likelihood over recently-published documents from Wikipedia, GitHub, BBC News, arXiv, bioRxiv, and AO3, as 15 category tasks plus a size-weighted group by @ziqing-huang in #3442 TyDiQA Gold Passage — 9-language multilingual QA by @bongho in #4044 LegalBench — new suite, 19 tasks: the HELM-lite LegalBench subset (5 tasks, #3860) and the Contract NLI suite (14 NDA entailment tasks, #3954) by @bongho IndicParam — MCQ benchmark covering 12 low-resource Indic languages by @hjoshifonteva in #3826 IndicXNLI Gujarati (indicxnli_gu) by @bhaumik611 in #4056 GreekMMLU — official native-sourced configuration by @mersinkonomi in #3581 Physics GRE — InflectionAI multiple-choice physics benchmark by @bongho in #3853 Putnam Axiom — competition-level mathematical reasoning by @baberabb in #3998 Portuguese Bench — ASSIN2 RTE and STS by @bongho in #3812 Catalanbench — cieaCOVA (#4043) and Terretaqa (#3784) by @baberabb and @ivanmartinezmurillo MLQA — a tag to run all variants at once by @ivanbaldo in #2977 Task Changes Correctness & Prompts global_piqa restructured into parallel/non-parallel × cloze/generation variants; the global_piqa_completions and global_piqa_prompted groups are now global_piqa_cloze and global_piqa_generation by @baberabb in #3816 IrokoBench — repaired afrimmlu / afrimgsm / afrixnli task registration by @discobot in #3841, fixed the broken afrisenti / mafand prompt_2 group references by @DaoyuanLi2816 in #3847, and switched afrixnli prompt_1 doc_to_text to Jinja braces by @Solaris-star in #3944 KorMedMCQA — answer-choice extraction now follows the paper's Appendix B by @discobot in #3842 JSONSchema Bench — per-sample validation timeout so a pathological schema no longer hangs the whole eval by @hancheolcho in #3923 RULER — every task crashed with TypeError: unhashable type: 'dict' before inference when the tokenizer arrived as anything but a plain string (e.g. under local-chat-completions); the name is now resolved before the cached lookup by @nata2627 in #4048 NIAH — no longer loops forever when max_seq_lengths < 4096 by @vnayakde in #3372 LongBench — code_sim_score skips a blank leading line before extraction by @cameronshinn in #3921 FDA / SWDE / SQuAD_completion — whitespace stripped from input and target by @EphraiemSarabamoun in #3795 med_prescriptions — requires both keys before combining complaints and diagnosis by @Anai-Guo in #4041 HumanEval — humaneval_random_span_infilling_light was registered under the wrong task name, causing a KeyError by @jaydeepborkar in #3768 INCLUDE — stray space removed from North Macedonian task identifiers by @DaoyuanLi2816 in #3848 DarijaBench — trasnlation typo corrected in translation task names by @DaoyuanLi2816 in #3824 Unitxt — dataset_kwargs is now passed through (#3230) and init sets the task name (#3225) by @mprahl format_span filter — normalizes labels only, leaving entity text untouched by @k-dickinson in #3887 ContextSampler — no longer crashes when the few-shot pool contains duplicate eval-doc rows by @Kymi808 in #3790 fewshot_config.split — a nested fewshot_config.split now takes precedence over the inherited top-level fewshot_split, as documented by @chuenchen309 in #3937 Model Backends & APIs Chat completions now support think_end_token by @yaodong-shen in #3959 auto:N batch size is parsed correctly in API models by @AbdullahRasheed45 in #3970, and batch_size="auto" no longer raises on the neuronx backend by @AbdullahRasheed45 in #3971 Synchronous API requests honor the configured timeout by @SiavashShams in #3995 Anthropic stop sequences are kept non-empty by @he-yufeng in #3822 HF: max_length is detected from a nested text_config (Gemma3 multimodal) by @OrionArchitekton in #3916, max_cpu_memory is passed through to accelerate's max_memory by @Anai-Guo in #4016, and tokenizers that decode to an empty string fall back to tokenizer.eos_token by @ganeshr10 in #3657 vLLM warns when it ignores the device argument by @Apeironics in #3803; sglang argument passing fixed by @wm901115nwpu in #3817 Core, CLI & Config --metadata accepts key=value pairs in addition to JSON by @baberabb in #4054 --use_cache keys are hashable for multimodal image/byte requests by @feiiiiii5 in #4040 Cache parent directories are created when missing by @sdivyanshu90 in #4047; delete_cache is a no-op when the directory is absent by @feiiiiii5 in #4035 A local directory sharing a task's name no longer shadows the registered task by @nloughl in #3670 Function resolving works for custom tasks by @SkyR0ver in #3992; string dictionary arguments parse correctly by @tandede in #4020 A scalar seed from a config file is normalized the same way the CLI does by @winklemad in #4003 Sample counts no longer depend on metric ordering by @arthi-arumugam-git in #4068 scripts/requests_caching.py entrypoint repaired — bad kwargs and a helper defined after its use by @Anai-Guo in #4059 chrF++ aggregation and metric (word_order=2) by @KrishVenky in #3780; TER metric direction corrected and the chrF docstring fixed by @borgr in #3993 All numpy scalar types serialize to JSON, not just int64/int32, by @iamsharduld in #3885 Migration Notes Few-shot prompts changed for tasks using a document-specific gen_prefix (RULER niah_single_1, humaneval_instruct, humaneval_64_instruct) and anywhere the sampler previously drew the eval document into its own shots. Prior numbers on those tasks may not be comparable. minerva_math and leaderboard_math_* now diverge. lm_eval/tasks/leaderboard/math/utils.py is deliberately frozen to reproduce Open LLM Leaderboard v2 scoring; the normalization fixes landed in minerva_math only. global_piqa task names changed — see Task Changes above. If you pinned datasets<4 to keep script-based tasks working, you can unpin. Out-of-tree task YAMLs pointing at script-based datasets still need the same treatment. winml now sits on top of onnxruntime-genai and pulls in that extra. What's Changed feat: 0.4.13.dev0 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3764 update global piqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3816 fix for keyerror while running humaneval_infilling by @jaydeepborkar in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768 fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks by @EphraiemSarabamoun in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795 Fix "trasnlation" typo in DarijaBench translation task names by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824 fix(ruler): replace defunct hotpot dataset host with pinned HF mirror by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3806 fix: keep Anthropic stop sequences nonempty by @he-yufeng in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822 fix(vllm): warn when device argument is ignored by @Apeironics in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803 fix sglang args by @wm901115nwpu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817 feat(legalbench): add HELM-lite LegalBench subset (5 tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860 Fix space in North Macedonian task identifiers (INCLUDE suite) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3848 fix(kormedmcqa): extract answer choice per paper Appendix B by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842 Fix dataset paths for
Authors: Lintang Sutawika, Leo Gao, Hailey Schoelkopf, Stella Biderman, Jonathan Tow, Baber Abbasi, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, Julen Etxaniz, Zdeněk Kasner, Khalid, Jeffrey Hsu, AndyZwei, Pawan Sasanka Ammanamanchi, Dirk Groeneveld, Ethan Smith, Eric Tang
Language: English
Open Access status: green
OpenAlex ID: W6893809404
Cites:
Cited by: 916
Related to: 9