Taught to talk
After its first round, AI readers picked her reply as the one most like a person’s on 80% of openings — the 4B open model on 18%, her base model on 0%
The first round learned from 102,673 conversations from openly licensed sets, most of them written by people, and our own answer set — none of the conversations we generated. Then our own generated conversations began to pass the bar: four sets of them, each read by two AI readers (70 of 80, 72 of 80, 68 of 80 and 70 of 80 read good), joined the next three rounds, until 5,066 of the 107,739 conversations were ours. Over every batch read, the recipe that made most of them reads 83% good (332 of 400, five batches): a set passes partly by the luck of its sample, so the pooled figure is the recipe’s own.
A preference step — learning which of two replies people preferred, from 10,253 pairs that people ranked — ranked held-out pairs the people’s way only 59.3% and 59.7% of the time, barely better than a coin. Round 4, without it, was chosen on 8 November: the AI readers preferred it in English openings, the one place its interval cleared the others’ (only just), and leaned the same way over several turns; in Romanian the three were level.
- Her larger model, first round — English (37)81%
- Her larger model, first round — Romanian (23)78%
- The 4B open model — English (37)16%
- The 4B open model — Romanian (23)22%
- Her base model, never taught to talk — English (37)0%
- Her base model, never taught to talk — Romanian (23)0%
- English openings (37) · Round 446%, 95% interval 30 to 62%
- English openings (37) · Round 3 with the preference step16%, 95% interval 5 to 30%
- English openings (37) · Round 4 with the preference step16%, 95% interval 5 to 30%
- Romanian openings (100) · Round 433%, 95% interval 24 to 42%
- Romanian openings (100) · Round 3 with the preference step33%, 95% interval 24 to 42%
- Romanian openings (100) · Round 4 with the preference step32%, 95% interval 24 to 41%
- Several turns (50) · Round 442%, 95% interval 28 to 56%
- Several turns (50) · Round 3 with the preference step24%, 95% interval 12 to 36%
- Several turns (50) · Round 4 with the preference step24%, 95% interval 14 to 36%
How we know
AI readers — fresh instances of an AI model, each given only the brief and its own items — saw the replies with the systems hidden and the order shuffled, and picked the one that sounded most like a real person talking, or called a tie. Long replies were cut short for the reading. Every figure here is an AI model’s judgement; no person has read these replies yet. The same family of AI model set the bar our own conversations passed, so part of round 4’s lead may be the reader’s taste.