Amazon has announced a new research challenge titled Knowledge-Grounded Task-Oriented Dialogue Modeling on Spoken Conversations, which aims to improve the robustness of conversational AI in realistic scenarios involving noisy speech inputs. The challenge has been accepted as a track in the 10th Dialog System Technology Challenge (DSTC10), the latest instance of the leading research challenge for advancing dialogue systems.
The Amazon challenge has two tracks, dialogue state tracking and knowledge-grounded task-oriented conversational modeling. In both cases, the goal is to develop models that work well when the input is a real speech signal, not just text.
Dialogue state tracking is one of the most widely studied problems in the dialogue research community. It involves estimating and tracking the customer’s goal throughout a conversation.
Amazon introduced knowledge-grounded task-oriented conversational modeling at last year’s DSTC. There, the goal is to use online information to supplement the information available through a specific application programming interface. For instance, a dialogue-based hotel-booking agent will have access to information like room rates and availability. But if a customer booking a room asks the cost of parking at the hotel, the agent may have to retrieve that information from an FAQ on the hotel’s site.
Most work on both these problems has used public data sets that consist only of written conversations, which leaves a gap between the resulting models and practical use cases involving spoken inputs.
In the new Amazon challenge, participants will develop dialogue systems for either or both tasks using any public data, most of which still consists of written conversations. The final evaluation, however, will use spoken data, encouraging participating teams to focus on building robust systems.
“The goal of this challenge is to bridge the gap between academic research and practical applications,” says Seokhwan Kim, a senior applied scientist in the Amazon Alexa AI organization. “We hope to inspire algorithms for more robust conversational systems in practice, something that previous challenges and datasets don’t address.”
The DSTC has been organized annually since 2011. Last year, Amazon’s track, Task-Oriented Conversational Modeling with Unstructured Knowledge Access, was the most successful track, with 105 competing systems submitted by 24 teams.
This year, Amazon’s proposal has again been selected as one of DSTC’s four main tracks. Challenge entry will remain open until September 21, and research teams from academia, industry, and nonprofit and government sectors are welcome to participate. Amazon has open-sourced the development data, evaluation scripts, and baseline systems for challenge participants and other researchers in the field.
