What you need to know
- The Brain Team at Google Research outlined a new type of chatbot in a paper published earlier this week.
- Named Meena, this virtual talker was explicitly designed to address chatbots' seeming inability to understand the broader context of conversations.
- It can also be funny, sometimes.
Meena, what Google Brain's research team so callously described as a "multi-turn open-domain chatbot" — she's got feelings, people! — could just be the next generation of chatbot. And perhaps your next best friend.
Unlike Samsung's awful human facsimiles, this Google-made AI chatterbox looks like it could probably hold its own in a conversation with a real human (via 9to5Google).
The paper describes the flaws with the current generation of chatbots as follows:
they often respond to open-ended input in ways that do not make sense, or with replies that are vague and generic.
Trained on 40 billion words mined from the valleys of Twitter and Facebook, Meena was explicitly designed to be contextually aware of not just some specific customer support tasks but rather the ongoing conversation as a whole and the world at large. To ensure Meena was smarter, and funnier, than those 'other chatbots,' Google's team coined two new metrics for testing itsperformance.
These being perplexity, which automatically generates data on the ability of a chatbot to understand a conversation, and Sensibleness and Specificity Average (SSA), which relies on humans to give their input on how well the AI performs in a conversation.
If anything seems off — confusing, illogical, out of context, or factually wrong — then it should be labeled as, "does not make sense".However, being sensible is not enough. A generic response (e.g., I don't know) can be sensible, but it is also boring and unspecific.For example, if A says, "I love tennis," and B responds, "That's nice," then the utterance should be marked, "not specific". That reply could be used in dozens of different contexts. However, if B responds, "Me too, I can't get enough of Roger Federer!" then it is marked as "specific", since it relates closely to what is being discussed.
In other news, I'm more boring than a chatbot, because all my conversations end with a "That's nice."
Getting back to Meena, what's the end result of all that geeky end-to-end neural network training? Meena is not only far more human-like in its conversations than its counterparts, but she's also funny.
And a budding philosopher, apparently.
That joke, by the way, was all Meena and officially makes her funnier than me. Whoever said chatbots are dumb and robotic?
Overall, Google was able to hone Meena to an SSA rating of 79%. That's particularly impressive given that it's not only 23% better than its closest competitor but, more importantly, it's also within spitting distance of the 86% score that was associated with humans.
If you'd like to get into all the nitty-gritty of how Google made Meena, you can read the 38-pages-long paper here.
Google's Sundar Pichai says we should be 'clear-eyed' about the dangers of AI, calls for regulation
