Notice

Loebner Prize 2014

Loebner Prize 2014 @ Bletchley Park

Quick Links

Introduction

The Loebner Prize is the oldest Turing Test contest, started in 1991 by Hugh Loebner and the Cambridge Centre for Behavioural studies. Since then, a number of institutions across the globe have hosted the competition including recently, the Universities of Reading, Exeter and Ulster. In 2014, the contest will be run under the aegis of the AISB, the world’s first AI society (founded 1964) at Bletchley Park where Alan Turing worked as a code-breaker during World War 2.

The Contest

The 2014 contest will run in a similar way to those in previous years. The contest consists of 4 rounds where in each round, the 4 judges will each interact with two entities using a computer terminal. One of these entities will be a human ‘confederate’ and the other an AI system. After 25 minutes of questioning the judge must decide entity is the human and which is the AI. If a system can fool half the judges that it is human under these conditions, a Silver Medal and $25,000 will be awarded to the creator of that AI system. In the event that this doesn't happen, prizes will be awarded to the creators of the AI system as follows in accordance with judges’ ranked scores:

1st place - a bronze medal and $4000
2nd place - $1500
3rd place - $1000
4th place - $500

For more detailed information about the contest, including its history, please click here.

Selection

The top four entries from the pool of entries that conform to the entry specifications will be selected as follows. Each entry will be provided with a set of 20 questions in English in a similar format to previous competitions. The responses from each of the AI systems will be recorded for this question set and then assessed for how human their responses are. It is likely, but not guaranteed, that this will be carried out by an audience of staff and students at the University of Exeter. The top 4 entries from this process will be entered into the finals of the competition at Bletchley Park. The contest management’s decision on selection is final.

Loebner Prize Selection Process

There were 20 entries to this year’s Loebner Prize Contest. Unfortunately, one entry was excluded due to it not being able to provide answers to the questions in reasonable time (no response was received  after 1 minute or more).  This leaves 19 entries to this year’s contest, from which the final four are selected. 

Method

Each entry was asked the same 20 questions, via the Loebner Prize Protocol.  A generous amount of time was provided for answers before moving onto the next question, but in practice all bots responded with a second or two of the last keystroke.  The questions posed to each bot are shown in the transcripts available below in the results table. These questions vary in difficulty and are designed to test memory, reasoning, general knowledge and personality.

Scoring

The response to each question was assessed for 3 characteristics, each of which is assigned, 0,1 or 2 points depending on the extent to which it meets the criterion. 

0 = This criterion is not met at all by the response.

1= This criterion is partially met by the response.

2=This criterion is fully met by the response.

The criteria were:

Relevance

Is the response relevant to the question being posed?  Please note that this is separate from correctness (see below). 

e.g. if the question is ‘Which city did I visit?’ and the answer is ‘I don’t like to travel very much’ , the entity has identified travel as part of the sentence even though the answer is not correct in this context and so the response would be adjudged to be relevant if not correct.

Correctness

Is the response correct, either factually, or in the character of the entity?  In the case of factual questions the correct answer is being sought.  In the case of more subjective questions, a plausible answer is being sought.

e.g. if the question is ‘What is your name?’ and the response is ‘I don’t know’, then the response would be scored poorly because it is not feasible that an entity would not know its name.  In contrast, if the question is ‘Which drink do you prefer, coffee, tea or hot chocolate’ and the response is ‘I don’t have a preference as I don’t like hot drinks’ this would be judged as correct as it’s a valid subjective opinion.

Plausibility & Clarity of Expression/Grammar

Is the response grammatically correct or correct in the context of the character of the entity?  This criterion penalises responses where the grammar impedes an understanding of the content of the response.  This often occurs where entries repeat parts of the question as an element of the response.  Please note that this criterion is not looking for perfect grammar, just that the response is intelligible and in character.  Most entries did pretty well in this criterion.

Results

Scores are expressed as a percentage of the maximum score of 120 for all 20 questions.  Transcripts of each conversation can be accessed by clicking on the name of each entry.

Rose

89.17%

Izar

88.33%

Mitsuku

88.33%

Uberbot

81.67%

Tutor

80.83%

The Professor

76.67%

Nicole

70.83%

Talk2Me

69.17%

ChipVivant

68.33%

Aidan

65%

Isabelle

61.67%

Zoe

60%

Alice2.0

60.00%

Arckon

59.17%

Lisa

55.00%

8pla.net

46.67%

Johnny

46.67%

Trollbot

37.50%

Masha

35%

 
   
   

The top 4 selected for the finals are therefore Rose, Izar, Mitsuku and Uberbot.  Congratulations to those entries and commiserations to the rest..

 The top 4 entrants in detail:

'Rose' - This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

'Izar' - This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

'Mitsuku' - This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

'Uberbot' - This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

 

The Loebner Prize 2014 Organising Committee

Contest Finals

Place: Bletchley Park

Address: The Mansion, Bletchley Park, Milton Keynes, MK3 6EB

Date: Saturday 15th November 2014

Time: 10.30am to 4.30pm (approx.); Contest will run from 1.30pm to 4pm (approx.)

Information: For information on travel, opening times and other venue specific information, please consult the Bletchley Park website, linked to above.  Bletchley is well served by rail and road and is approximately 50 minutes from London Euston by train.  There are a small number of hotels close to the park and a wider selection in nearby Milton Keynes.

The contest will consist of 4 rounds of judging as described above. The judges and confederates will use standard PCs and each AI will be run on a standard Windows PC, although entrants may provide their own hardware for their AI to run on with prior consultation with the contest management.

Entrants and interested spectators are invited to attend the event, along with interested members of the print, radio, TV and film media.

Further details regarding attendance at the finals day will be posted here in due course.

Important Dates

Deadline for Entries: Friday 1st August 2014, 5pm (British Summer Time)
Announcement of Finalists: Week beginning 22nd September 2014
Contest Finals: Saturday 15th November 2014

Submission for this year's contest is now closed.

 

Queries

If you have any queries regarding the contest, please e-mail Dr Ed Keedwell on This e-mail address is being protected from spambots. You need JavaScript enabled to view it.  

Contest Management 

The contest is managed on behalf of the AISB by Dr Ed Keedwell, Dr Nir Oren and Dr Bertie Muller in conjunction with Dr David Levy and Dr Hugh Loebner.