Young researchers face-to-face on human-machine dialogue
David Pardo, Milica Gašić, Joana Paulo Pardal, Ricardo Ribeiro, Matthew Marge and François Mairesse
October 2009
The fifth Young Researchers' Roundtable on Spoken Dialogue Systems (YRRSDS'09) was held at Queen Mary University in London last month. The event - organized by young (and not so young!) researchers, most of whom were participants in the 2008 edition in Columbus, Ohio - brought together 41 researchers in academia and industry, from a wide variety of institutions and from several continents. Many had also participated in the SIGDial or Interspeech conferences with which the event is affiliated.
The first roundtable was organized in Lisbon, Portugal in 2005 by the "Dialogs on Dialogs" group based at Carnegie Mellon University. Its goal was to provide dialogue system researchers in the early stages of their careers an opportunity to engage in multidisciplinary discussions - this being the nature of the field -, to meet other researchers and be in touch with other research efforts, and to build bridges between the realms of academia and industry.
This year's edition was organized in the same spirit. It featured talks by senior researchers in industry and academia and special sessions on career advice and the "grand challenges" of evaluating the performance of, and the user's experience with, dialogue systems. A demo and poster session also allowed some of the participants to present their work. All of the participants had previously submitted a position paper describing their current research and topics of interest. The proceedings of the event are available at www.yrrsds.org.
Invited Speakers
Among the attendants were seven invited speakers. Philippe Bretier outlined a new methodology that is being used at Orange (formerly France Telecom) for modelling dialogue based on reinforcement learning. Tim Paek spoke of his personal motivations to work in Microsoft Research and contrasted them to what he misses from the academic world. Alan Black (Carnegie Mellon University), Michael McTear (University of Ulster) and Jason Williams (AT&T Labs - Research) shared their career experiences and offered advice to young researchers seeking to find their way in academia or industry. Michael McTear stressed the importance of publishing in high quality journals and conferences, and pointed out that young researchers should be aware of reviewing criteria and also be prepared to resubmit their work. Alan Black discussed how making progress in academia can work differently from country to country, and spoke of his own experience combining work in both academia and industry. Jason Williams spoke of the relative importance of different factors in deciding where to seek employment, such as the people with whom one will work (very important) and location (not important).
Sebastian Möller (Deutsche Telekom Labs) was the first to approach the evaluation theme. Sebastian addressed the issue of automatic evaluation of the quality and usability of spoken dialogue systems, and how the problem is handled differently in academia and industry. Dan Bohus demonstrated the complexity of the problem of evaluation by presenting a list including many of the main metrics that have been proposed in recent years, from 'number of dialogue turns' to 'F-score for definite exophoric reference resolution'. He pointed out, as Möller had before him, that performance metrics are not "quality" from the point of view of the user's experience. Dan proposed addressing such complexity by performing local evaluations on particular aspects, although questions of significance to the whole could then arise. Finally, Alan Black captured the imagination of the attendees when he invited them to take part in the Spoken Dialogue Challenge: the Let's Go system developed at the Language Technologies Institute at CMU will be provided to the participants so that they can replace any module with their own and thus develop improvements in specific areas to be assessed with real users. Selected systems will run live in Pittsburgh so that real users can interact with them.
YRR group discussions
Discussion sessions are the heart of the roundtable. Participants split into groups to discuss subjects of interest at greater length. The more popular topics dealt with data collection and analysis, learning systems and user-system (mutual) adaptation, which may be an indication of some of the current trends in dialogue research. The difficulty (and perhaps even undesirability) of developing standards for the development of corpora was identified, as well as ethical barriers to the collection and sharing of data from test users. Tim Paek, SIGdial president, brought forward a proposal to have research groups gather together their data in a variety of domains (including human-human interaction) and make it available so that collaborative efforts of annotation, classification and perhaps further processing may be undertaken.
The trend towards a greater focus on the user was apparent throughout the event. The general opinion was that systems should be sensitive to context, particularly to the users' personal and changing needs (informational, cognitive and even emotional) throughout the interaction. It was widely suggested that knowledge of human-human interaction should guide the development of human-machine interaction systems, especially those with multi-modal capabilities. The importance of evaluating dialogue with real users was stressed. It was noted that it is not at all trivial to measure user satisfaction, but emotions and level of engagement were suggested as promising indicators. The peculiarities of multi-language and multi-domain dialogue systems were also discussed.
YRRSDS 2010...
All in all, for the fifth year in a row, the roundtable has proven to be a nice platform for sharing knowledge and ideas among young researchers in a friendly atmosphere. The organizers are now passing the torch for the next year's Young Researchers' Roundtable, which will take place in Japan!
Acknowledgements
Thanks to Christine Howes, Arash Eshghi and Gregory Mills for their great help with organising this event.
Add A Comment