{"id":19399,"date":"2022-05-24T07:29:08","date_gmt":"2022-05-24T14:29:08","guid":{"rendered":"https:\/\/www.kith.org\/jed\/?p=19399"},"modified":"2022-05-24T07:29:08","modified_gmt":"2022-05-24T14:29:08","slug":"apparently-im-gullible-when-it-comes-to-ais","status":"publish","type":"post","link":"https:\/\/www.kith.org\/jed\/2022\/05\/24\/apparently-im-gullible-when-it-comes-to-ais\/","title":{"rendered":"Apparently I\u2019m gullible when it comes to AIs"},"content":{"rendered":"\r\n<p>Deepmind has created a new <a href=\"https:\/\/www.deepmind.com\/publications\/a-generalist-agent\">\u201cgeneralist agent\u201d AI<\/a>, known as Gato. \u201cThe same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.\u201d<\/p>\r\n<p>\u2026Which is neat, but the reason I\u2019m posting about it is the set of text-chat examples shown in the post.<\/p>\r\n<p>Among other things, Gato tells its interlocutor the following:<\/p>\r\n<ul>\r\n  <li>J.G. Ballard\u2019s book <cite>Canyon Song<\/cite> (which as far as I can tell doesn\u2019t exist in the real world) is about the Archangel Gabriel and the Archangel Lucifer, \u201cwho was invited into heaven to act as the angelic host for the Bible.\u201d Also, \u201cDodecahedron Assemblies was the publication I chose to read on the way to heaven.\u201d<\/li>\r\n  <li>Percy Jackson\u2019s <cite>The Lightning Thief<\/cite> \u201cis about a mysteriously injured, mostly deaf, man who loses his ears during a storm and discovers that he is able to close the exit holes on the buildings he lives in.\u201d (Some of the rest of the made-up description is arguably ableist, though I\u2019m not entirely certain how to interpret it. I initially wrote \u201cI\u2019m not entirely certain how Gato intended it,\u201d and then had to stop and remind myself that Gato didn\u2019t \u201cintend\u201d anything here.)<\/li>\r\n  <li>The name of AlphaFold (Deepmind\u2019s protein-folding software) \u201cis based on a roof tile of the University of Maryland, which matches the sequence number and neuroscience thread index, which is how neuroscience researchers manage their data.\u201d<\/li>\r\n  <li>The capital of France is Marseille. (No, it\u2019s Paris.)<\/li>\r\n  <li>Famous jazz musicians from Marseille include \u201cHerbie Hancock, John Coltrane, Tony Bennett, Etienne Wouter.\u201d (The first three are American, with no particular connection to Marseille that I can see. The last name does not appear to be the name of a famous jazz musician.)<\/li>\r\n<\/ul>\r\n<p>And I\u2019m fascinated both by these remarks of Gato\u2019s and by my reactions to them.<\/p>\r\n<p>At least half of the chat examples that the article\u2019s authors chose to include show Gato making firm, confident-sounding, and factually incorrect statements. (But some of the other statements are true.) Nothing in the human-written text of the summary post that I linked to above indicates that there\u2019s anything wrong with those responses; the post just shows that set of responses as one example of the many cool things that Gato can do.<\/p>\r\n<p>The paper itself (linked from the page that I linked to) does say \u201cUsually Gato replies with a relevant response, but is often superficial or factually incorrect, which could likely be improved with further scaling.\u201d So the authors were aware that Gato is just wrong in many of its answers, but that fact is irrelevant to what they\u2019re trying to do. Fair enough.<\/p>\r\n<p>But I also find my reactions interesting. Because when I read an exchange between a question-asking human and an information-providing AI system, apparently the format primes me to expect factually accuracy, especially when the responses are mostly grammatically correct and seem to be mostly on-topic. And especially when some of the responses are correct, and when others are on topics I\u2019m not familiar with so they seem like they could be correct.<\/p>\r\n<p>So as I read Gato\u2019s responses, without knowing that they were known to be incorrect, I got increasingly bewildered. I went from <i>Huh, a couple of Ballard books that I\u2019ve never heard of<\/i> to <i>Interesting, I had no idea that\u2019s what <cite>The Lightning Thief<\/cite> is about<\/i> to <i>Wait, isn\u2019t AlphaFold called that because of protein folding? What does it have to do with roof tiles?<\/i><\/p>\r\n<p>I kept <em>expecting<\/em> the responses to be true and sensical, so it took me a while to convince myself that several of them were false and\/or nonsensical.<\/p>\r\n<p>(Which is especially interesting to me because I\u2019m usually a very suspicious reader; when humans say stuff, I\u2019m often watching for misstatements. But apparently somehow this format lulled me into turning off the suspicious part of my brain. That\u2019s \u2026 not ideal.)<\/p>\r\n\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[79],"tags":[],"class_list":["post-19399","post","type-post","status-publish","format-standard","hentry","category-computers"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/posts\/19399","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/comments?post=19399"}],"version-history":[{"count":3,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/posts\/19399\/revisions"}],"predecessor-version":[{"id":19402,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/posts\/19399\/revisions\/19402"}],"wp:attachment":[{"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/media?parent=19399"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/categories?post=19399"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kith.org\/jed\/wp-json\/wp\/v2\/tags?post=19399"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}