Loebner Prize at InterSpeech 2009

Filip Jurcicek

SLTC Newsletter, January 2009

Since 1991, the Loebner Prize contest has tried to find a computer which is able to pass the Turing Test. The test was proposed by Alan Turing in his article "Computing Machinery and Intelligence" as a proxy for his original question "Can machines think?"

In the Loebner Prize’s implementation of the Turing Test, a human judge is engaged in a conversation with one human and one chatbot, a computer which mimics humans. The only mean of communication is a keyboard and a computer screen, and both the chatbot and the human must try to convince the judge that they are human. At the end, the judge must decide who responded more human-like.

Hugh Loebner offers three awards. First, there is an annual competition for the most human-like chatbot among contestants. The winner of the annual competition receives a bronze medal and a cash award of $3,000. This year’s annual competition will be held in conjunction with InterSpeech, in Brighton, UK, on 6 September 2009. Chatbots will interact with judges for 10 minutes each.

Hugh Loebner also offers a periodic competition for a silver medal. To win the silver medal, which includes a cash prize of $25,000, a chatbot must interact with judges for 25 minutes and convince 50% of the judges that the chatbot is human.

Once technology advances to an appropriate level, a competition for a gold medal grand prize will be organized. The grand prize, which includes a cash award of $100,000, will go to a chatbot which passes the wider Turing Test involving audiovisual input.

During the last annual Loebner Prize contest on October 12, 2008 at the University of Reading, five chatbots competed in the contest. The bronze medal was won by a chatbot named Elbot, which was developed by Fred Roberts from Artificial Solutions, in Germany. Elbot convinced three of the twelve judges that it was human, whereas the other four chatbots managed to convince at least one judge.

Although these chatbots are not perfect, many companies have already successfully deployed chatbots on their web sites. For example, you can find Valerie at Virgin Holidays, Jenn at Alaska Airlines, or Spike at Gonzaga University website. If you ask Jenn about flight from Boston to LA on the next Sunday, it will take you to the partially filled web page where you can specify the rest of the details of your trip and list available flights. Spike correctly answers questions such as "What can I study at the university?" and "How much does it all cost?" These chatbots manage to improve user experience by easing access to buried information in the websites.

As the theme of InterSpeech 2009 is 'Speech and Intelligence', the contest perfectly complements the conference. After all, the contest is about designing intelligent machines that perceive and respond using natural language. "The contest draws public attention to NLP field and public benchmarking has become one of the main drivers in the field, and a live test should be one of a battery of such tests," says Prof. Roger Moore, General Chair for InterSpeech 2009.

In addition to finding which chatbot is the most human-like, some other interesting findings can be observed. For example, Dr. Philip Jackson, responsible for the Loebner Prize contest at InterSpeech, expects to learn something new about how humans differentiate between human and machine conversation.

In any case, the contest promises a lot of fun! The imperfection of the chatbots is always amusing and there is nothing more pleasant than to know that humans are unlikely to be replaced with chatbots.