GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Year: 2022
Type: article
Abstract: Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work.The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices.We pose that the standardization is taking place in the wrong spot.Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances.We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work.GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.
Source: Greater South Information System
DOI: 10.60692/46kpf-9d950
Authors: Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Анна Швец, Ashish Datt Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Gârbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth A. Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo Sampaio Ferraz Ribeiro, Lewis C. Tunstall, Zhang Li, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondřej Dušek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Zhu Qi, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sébastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, В. А. Николаев, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
Institutions: Google (United States), Bangladesh University of Engineering and Technology, New York University, Amazon (United States), Carnegie Mellon University, Indian Institute of Technology Hyderabad, Robert Gordon University, Rensselaer Polytechnic Institute, Hong Kong University of Science and Technology, University of Hong Kong, Allen Institute, Tianjin University, University of Aberdeen, University of Michigan, Microsoft (United States), IBM (United States), California University of Pennsylvania, Edinburgh Napier University, Yale University, Stanford University, Columbia University, University of Turku, Salesforce (United States), nLIGHT (United States), Queen Mary University of London, Harvard University, University of California, Santa Cruz, Emory University, University of Edinburgh, Technische Universität Darmstadt, University of Helsinki, The Ohio State University, École Polytechnique, RWTH Aachen University, Charles University, Tsinghua University, Johns Hopkins University, Universitat Pompeu Fabra, University of Kaiserslautern, Luleå University of Technology, Syracuse University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6920598330
