SSP Forum: Christopher Potts on Benchmark NLP Datasets
The
Symbolic Systems Forum
presents
Benchmark datasets: The essential resources on which all NLP depends
Christopher Potts
Linguistics Department
Monday, February 14, 2022
12:15-1:15 pm
ABSTRACT:
Like many areas of AI, present-day NLP is data-driven. As a result, the available benchmark datasets are the primary factor in shaping the field itself. This has wide-ranging consequences for research, technology, and increasingly for society. How do we conceptualize different tasks, and which tasks receive the most attention from researchers? Which languages are adequately represented in our literature? Which groups benefit most from language technologies? Where will our systems deliver results that are embarrassing or worse? The answers to all these questions lie largely in the data we have for training and assessment. It is therefore in our best interests to deeply understand the datasets on which we are so dependent, and to seek out innovative new ways of collecting and validating relevant data. In this talk, I will report on a number of recent efforts to create more meaningful benchmarks for the field, and I will seek to identify persistent challenges and open questions in this area.
A NOTE ON THE RECORDING OF EVENTS:
If a decision has been made in advance to record an event and to make it available for later public viewing, the event announcement will usually state this. In many cases, however, decisions to record, and/or to make a recording available publicly, are not finalized before an event is announced. Availability decisions for recordings are often subject to what speakers prefer after an event has concluded, among other considerations that may include usage rights for material used in an event, as well as the need for, and practicality of, editing. When recordings are made publicly available, they will be linked within the original event announcement on the Symsys website in the days or weeks following an event. Unfortunately, we cannot follow up on individual requests for more information about whether and when a recording may become available if it is not yet posted publicly.