Lorem Ipsum

Deepmind has created a new “generalist agent” AI, known as Gato. “The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.”

…Which is neat, but the reason I’m posting about it is the set of text-chat examples shown in the post.

Among other things, Gato tells its interlocutor the following:

• J.G. Ballard’s book Canyon Song (which as far as I can tell doesn’t exist in the real world) is about the Archangel Gabriel and the Archangel Lucifer, “who was invited into heaven to act as the angelic host for the Bible.” Also, “Dodecahedron Assemblies was the publication I chose to read on the way to heaven.”
• Percy Jackson’s The Lightning Thief “is about a mysteriously injured, mostly deaf, man who loses his ears during a storm and discovers that he is able to close the exit holes on the buildings he lives in.” (Some of the rest of the made-up description is arguably ableist, though I’m not entirely certain how to interpret it. I initially wrote “I’m not entirely certain how Gato intended it,” and then had to stop and remind myself that Gato didn’t “intend” anything here.)
• The name of AlphaFold (Deepmind’s protein-folding software) “is based on a roof tile of the University of Maryland, which matches the sequence number and neuroscience thread index, which is how neuroscience researchers manage their data.”
• The capital of France is Marseille. (No, it’s Paris.)
• Famous jazz musicians from Marseille include “Herbie Hancock, John Coltrane, Tony Bennett, Etienne Wouter.” (The first three are American, with no particular connection to Marseille that I can see. The last name does not appear to be the name of a famous jazz musician.)

And I’m fascinated both by these remarks of Gato’s and by my reactions to them.

At least half of the chat examples that the article’s authors chose to include show Gato making firm, confident-sounding, and factually incorrect statements. (But some of the other statements are true.) Nothing in the human-written text of the summary post that I linked to above indicates that there’s anything wrong with those responses; the post just shows that set of responses as one example of the many cool things that Gato can do.

The paper itself (linked from the page that I linked to) does say “Usually Gato replies with a relevant response, but is often superficial or factually incorrect, which could likely be improved with further scaling.” So the authors were aware that Gato is just wrong in many of its answers, but that fact is irrelevant to what they’re trying to do. Fair enough.

But I also find my reactions interesting. Because when I read an exchange between a question-asking human and an information-providing AI system, apparently the format primes me to expect factually accuracy, especially when the responses are mostly grammatically correct and seem to be mostly on-topic. And especially when some of the responses are correct, and when others are on topics I’m not familiar with so they seem like they could be correct.

So as I read Gato’s responses, without knowing that they were known to be incorrect, I got increasingly bewildered. I went from Huh, a couple of Ballard books that I’ve never heard of to Interesting, I had no idea that’s what The Lightning Thief is about to Wait, isn’t AlphaFold called that because of protein folding? What does it have to do with roof tiles?

I kept expecting the responses to be true and sensical, so it took me a while to convince myself that several of them were false and/or nonsensical.

(Which is especially interesting to me because I’m usually a very suspicious reader; when humans say stuff, I’m often watching for misstatements. But apparently somehow this format lulled me into turning off the suspicious part of my brain. That’s … not ideal.)