Loebner Prize @ Bletchley Park
Entry Submission Deadline: Wednesday 1st July 2015
Announcement of Finalists: Monday 3rd August 2015
Finals Day: Saturday 19th September 2015
The Loebner Prize is the oldest Turing Test contest, started in 1991 by Hugh Loebner and the Cambridge Centre for Behavioural studies. Since then, a number of institutions across the globe have hosted the competition including recently, the Universities of Reading, Exeter and Ulster. From 2014, the contest will be run under the aegis of the AISB, the world’s first AI society (founded 1964) at Bletchley Park where Alan Turing worked as a code-breaker during World War 2.
The 2014 contest was run in a similar way to those in previous years. The contest consists of 4 rounds where in each round, the 4 judges will each interact with two entities using a computer terminal. One of these entities will be a human ‘confederate’ and the other an AI system. After 25 minutes of questioning the judge must decide entity is the human and which is the AI. If a system can fool half the judges that it is human under these conditions, a Silver Medal and $25,000 will be awarded to the creator of that AI system. In the event that this doesn't happen, prizes will be awarded to the creators of the AI system as follows in accordance with judges’ ranked scores:
1st place - a bronze medal and $4000
2nd place - $1500
3rd place - $1000
4th place - $500
For more detailed information about the contest, including its history, please click here.
The top four entries from the pool of entries that conform to the entry specifications will be selected as follows. Each entry will be provided with a set of 20 questions in English in a similar format to previous competitions, with at least 2 Winograd style questions. The responses from each of the AI systems will be recorded for this question set and then assessed for how human their responses are. The top 4 entries from this process will be entered into the finals of the competition at Bletchley Park. The contest management’s decision on selection is final.
- Only one entry per individual or team is permitted. Additional entries by an individual or team will be excluded.
- Entrants retain all intellectual property rights over their entries to the contest.
- Entries must be accompanied by simple instructions for use/installation.
- If selected, entrants are permitted to use their own machine in the contest finals, providing it adheres to the contest rules. Any such entries should be agreed with the contest management well in advance of finals day.
- Entries can be submitted by mail or electronically. Please see below for instructions for each:
- Entries must be received by 1st July 2015 to be counted
- Entries should be made on a readily accessible media type (e.g. CD, DVD, memory stick). If you want to submit an entry on non-standard media, please contact the contest management before sending your entry
- If you require your media to be returned then a self-addressed envelope with adequate postage should be included with your entry. We cannot guarantee to return media without this.
- Entries by mail should also be accompanied by covering letter that includes the name and contact details (including e-mail and telephone if possible) of the creator. Please indicate on this letter the name of your entry and whether you, as the creator, wish to remain anonymous in the selection process.
Details of a mailing address will be made available shortly.
- Electronic submission of entries will be possible and details of this process will be posted here in due course.
- Electronic submissions must be received by 5pm (BST) on the 1st July 2015 to be counted.
- All electronic submissions will require an accompanying e-mail that includes the name and contact details (including e-mail and telephone if possible) of the creator. Please indicate in this e-mail the name of your entry and whether you, as the creator, wish to remain anonymous in the selection process.
- Entries are also permitted on the entrant's own machine. Entries on machines must adhere to the same specifications as electronic or mail submissions, i.e. they must not connect to the internet, must work with the Loebner Prize Protocol and must not require extensive effort on the part of the management to run.
- Machines sent via post or courier are sent at the entrant's own risk.
- Any entrants wishing to personally attend and setup their machines for selection do so under the understanding that they will not be permitted to interact or remain with the machine during the formal selection process.
- Any shipping of machines or attendance at the selection process must be agreed well in advance of the submission deadline with the contest management.
- Entries must run on Windows 7.
- Entries can be zipped up using the standard windows zip program, but otherwise should be run by invoking a standard Windows executable or batch file.
- Entries must be self-contained and not require installation of third party software or libraries.
- Entries must interact with the published Loebner Prize Protocol (LPP).
- On startup, entries should provide a mechanism for specifying a conversation folder as per the LPP (e.g. either through a dialog box or other similar mechanism).
- Entries must not access the internet, no internet access will be provided during the selection process or contest finals.
- Any entries that do not conform to the above specifications are at risk of being excluded.
- Any entries that are found to contain viruses, worms, malware etc. will be excluded.
- The contest management will have a very limited time to get each entry working. It is each entrant’s responsibility to ensure that their entry works with the above specifications.
The contest is managed on behalf of the AISB by Dr Ed Keedwell, Dr Nir Oren, Dr Bertie Müller, Andrew Martin and Janet Gibbs in conjunction with Dr David Levy and Dr Hugh Loebner.
Loebner Prize Selection Process
There were 20 entries to this year’s Loebner Prize Contest. Unfortunately, one entry was excluded due to it not being able to provide answers to the questions in reasonable time (no response was received after 1 minute or more). This leaves 19 entries to this year’s contest, from which the final four are selected.
Each entry was asked the same 20 questions, via the Loebner Prize Protocol. A generous amount of time was provided for answers before moving onto the next question, but in practice all bots responded with a second or two of the last keystroke. The questions posed to each bot are shown in the transcripts available below in the results table. These questions vary in difficulty and are designed to test memory, reasoning, general knowledge and personality.
The response to each question was assessed for 3 characteristics, each of which is assigned, 0,1 or 2 points depending on the extent to which it meets the criterion.
0 = This criterion is not met at all by the response.
1= This criterion is partially met by the response.
2=This criterion is fully met by the response.
The criteria were:
Is the response relevant to the question being posed? Please note that this is separate from correctness (see below).
e.g. if the question is ‘Which city did I visit?’ and the answer is ‘I don’t like to travel very much’ , the entity has identified travel as part of the sentence even though the answer is not correct in this context and so the response would be adjudged to be relevant if not correct.
Is the response correct, either factually, or in the character of the entity? In the case of factual questions the correct answer is being sought. In the case of more subjective questions, a plausible answer is being sought.
e.g. if the question is ‘What is your name?’ and the response is ‘I don’t know’, then the response would be scored poorly because it is not feasible that an entity would not know its name. In contrast, if the question is ‘Which drink do you prefer, coffee, tea or hot chocolate’ and the response is ‘I don’t have a preference as I don’t like hot drinks’ this would be judged as correct as it’s a valid subjective opinion.
Plausibility & Clarity of Expression/Grammar
Is the response grammatically correct or correct in the context of the character of the entity? This criterion penalises responses where the grammar impedes an understanding of the content of the response. This often occurs where entries repeat parts of the question as an element of the response. Please note that this criterion is not looking for perfect grammar, just that the response is intelligible and in character. Most entries did pretty well in this criterion.
Scores are expressed as a percentage of the maximum score of 120 for all 20 questions. Transcripts of each conversation can be accessed by clicking on the name of each entry.
The top 4 selected for the finals are therefore Rose, Izar, Mitsuku and Uberbot. Congratulations to those entries and commiserations to the rest..
The top 4 entrants in detail:
The Loebner Prize 2014 Organising Committee
Place: Bletchley Park
Address: The Mansion, Bletchley Park, Milton Keynes, MK3 6EB
Date: Saturday 15th November 2014
After 2 hours of judging at Bletchley Park, 'Rose' by Bruce Wilcox was declared the winner of the Loebner Prize 2014, held in conjunction with the AISB. The event was well attended, filmed live by Sky News and the special guest judge was revealed to be none other than television presenter and broadcaster James May. Bruce will receive a cheque for $4000 and a bronze medal. The ranks were as follows:
Rose - Rank 1 ($4000 & Bronze Medal)
Izar - Rank 2.25 ($1500)
Uberbot - Rank 3.25 ($1000)
Mitsuku - Rank 3.5 ($500)
Although the judges were unanimous in their view of Rose as the best entry, none of the entries fooled the judges, meaning that the Silver Medal and $25,000 dollar prize are still to be won.
The contest management would like to thank everyone involved in the organisation of the contest including the judges & confederates (listed below), Claire Urwin and Katherine Lynch of Bletchley Park and Paul Sant and Ray of the University of Bedfordshire.
Dr Ian Hocking, Writer & Senior Lecturer in Psychology, Christ Church College, Canterbury
Dr Ghita Kouadri-Mostefaoui, Lecturer in Computer Science and Technology, University of Bedfordshire
Mr James May, Television Presenter and Broadcaster
Dr Paul Sant, Dean of UCMK, University of Bedfordshire
- The day was filmed for Sky News, both for their standard news broadcast and for their DigitalView program (Turing piece from approximately 15:20).
- One of the judges, Dr Ian Hocking, has written a blog about his experiences here.