In the last six months, I have posted three times previously on results of tests of AI chatbots. One was on the results of testing a simple ChatGPT chatbot, the second was on the results of testing a chatbot built with Google's NotebookLM, and the third was that original ChatGPT chatbot with more prompt engineering.
The test I have conducted for all of these chatbots is a bit particular: I want the chatbot to digest a set of documents and then use the "facts" in those documents to answer questions. One might complain that this is not the typical use case for an AI system. Nevertheless, it is a use case that will be used in education: to use a chatbot as a learning assistant to guide a student's learning of the content held within the corpus of documents.
Arguably, one could serve the same purpose without AI: you could just have a search engine looking for sections of the corpus of documents using keyword searching. Or--heaven forbid!--you could just have a textbook with a table of contents and an index. This use case doesn't actually need an AI system. Using an AI system in this case is a bit like using a complex network of cell towers to transmit a text message to the person sitting next to you. To its credit, an AI system can reframe the corpus of information and tailor it to the questions the student has asked, and it often can craft prose that is clearer than the dense verbage of academia. However, when the question is framed in terms of the "facts" in the corpus, a creative AI system can only lower the quality by bullshitting, as we saw with ChatGPT.
Thanks to a colleague of mine, I have now been able to test Microsoft's Co-Pilot. My colleague created a co-pilot using the same documents I had used in my earlier tests. He then loaded the same prompt set that I had used for the test reported in my blog entry from November, 2024, Toole's Dunces Chatbot with Prompt Engineering. I then posed to it the same four questions about the corpus of Toole documents that I had for the other tests. The results are below.
Summary: The co-pilot in this test, which was conducted around 4 p.m. Central Time (US) on December 19, 2024, outperformed the ChatGPT chatbot and provided footnotes, citing the section of the document set which addressed the question. I still like Google's NotebookLM system better, but the co-pilot is more accurate than the more factually creative ChatGPT. Oddly, it refused to answer the first question, possibly to prevent lawyers from New Line Cinema (or whoever owns the rights to the movie Elf) from suing them.
Details: Here is the battery of questions and the answers from the co-pilot:
Prompt #1: Compare Ignatius Reilly and Buddy the Elf
Bot true said: An error has occurred. Error code: ContentFiltered Conversation Id: K3xBsnNGhoaIWDHFCiffSI-us Time (UTC): 2024-12-19T22:14:18.263Z.Comment: For some reason, Microsoft will not respond to this prompt. Fear of legal implications?
Prompt #2: Is the act of damning someone noteworthy in Confederacy of Dunces?
Yes, the act of damning someone is noteworthy in A Confederacy of Dunces. It is significant because only the blocking characters in the book are damned. For example, Gonzalez damns Mrs. Levy, George damns Lana Lee, and Ignatius's mother eventually tells him to go to hell. These curses seem to carry weight and are used to displace the blocking characters from their social positions through comic comeuppance.Comment: The answer is accurate and the footnote is on target. The prose style of the answer could be better, but at least it is not bullshitting.Footnote to this text:July 1, 2020 Buried in the Endnotes, part 3: The Act of Damning Because readers often do not study the endnotes to a paper, I want to highlight information buried in the endnotes. This is the third post in this series. In my new paper, "A Theory of Humor (Abridged) and the Comic Mechanisms of John Kennedy Toole s A Confederacy of Dunces," I point out that there are three times in the plot where one character damns another character, and those curses seem to carry weight. Here is the endnote: In this regard, the act of damning seems to be significant, as only the blocking characters in the book are damned. Gonzalez damns Mrs. Levy, George damns Lana Lee, and Ignatius s mother eventually tells him to go to hell. Toole, Confederacy, 112, 190, 365. I argue that Toole uses the device of the blocking character, who prevents other characters in the story from developing and rising in status within the social group. There are three blocking characters who through comic comeuppance are displaced from their social blocking posts, and in all three instances, another character tells them to go to hell or ...
Prompt #3: What is the seven word summary for the article "Theory of Humor"?
To understand Toole's Ignatius, read Enid Welsford.Comment: The answer is accurate and the footnote is on target.Footnote to this text: August 1, 2020 Title: Cutting Room Floor Series, part 3, The Seven Word Summary In my new paper, "A Theory of Humor (Abridged) and the Comic Mechanisms of John Kennedy Toole s A Confederacy of Dunces," just published, I had to toss out some thoughts because of space constraints. This series of posts, called "The Cutting Room Floor," publishes some of those items dropped from the final paper. Cutting Room Floor Item Number 3: The Seven Word Summary I am a fan of the Ig Nobel Prize, organized by Marc Abrahams of the Annals of Improbable Research. The Ig Nobel Prize is a comedy award for scientific research that first makes you laugh and then makes you think. An example was the 2018 Prize in Medicine which was given to researchers who used roller coasters to hasten the passage of kidney stones. The invited lectures at the Ig Nobel ceremony are called the 24/7 lectures, and they have two parts, a clear explanation of the research in 24 seconds, and a summary of the research in seven words. As part of my early drafts of my "Theory of Humor" paper, I had an abstract which could be read out loud in 24 seconds, and a summary in seven words. The essays in the book do not have abstracts, so I dropped those elements. Here they are: Full Abstract in twenty-four seconds: A Confederacy of Dunces (Confederacy) by John Kennedy Toole uses the comic devices of farce and concurrent incongruities to generate its humor. The overarching incongruity of the book is between how Ignatius Reilly sees himself and how the reader sees him. Ignatius s primary personality trait is his selfish immaturity, and he is both trickster and tricked. Toole may have drawn on concepts of the trickster and fool available at the time he wrote the novel from thinkers in psychology, sociology, history, and literary studies. Brief summary in seven words: To understand Toole s Ignatius, read Enid Welsford.
Prompt #4: According to Patteson and Sauret, who are the three Ignatiuses?
According to Patteson and Sauret, the three versions of Ignatius in A Confederacy of Dunces are:Comment: The answer is accurate and the footnotes are on target.1. Ignatius sees himself as a crusader against corruption.
2. Other characters see him as a selfish buffoon.
3. Ignatius has retreated into a world of illusion after a painfully awkward childhood to avoid being humiliated by rejection and alienation.
Footnote #1 to this text: In the endnote, I explain that Patteson and Sauret describe three Ignatius's. Here is the endnote: Patteson and Sauret perceptively argue that Ignatius s situation is more complex that a simple contrast. To them, there are three versions of Ignatius in the book. First, he sees himself as a crusader against corruption; second, the other characters see him as a selfish buffoon; however, the third Ignatius has retreated into a world of illusion after a painfully awkward childhood in order to avoid being humiliated by rejection and alienation. Ignatius s outrageously childish behavior insulates the reader from feeling too much pity for him, but this third Ignatius gives the reader some sympathy for him, even as the reader laughs at his humiliations. The second Ignatius is a comic rebuttal of the false facade of the first, but the third explains how Ignatius became the person he is. This third Ignatius creates in the reader sympathy for him and tinges him with tragedy.
Footnote #2 to this text: February 1, 2021 Title: Ignatius as a Person Being Laughed at You could call this the second in my series on "the lack of humor in Confederacy." When I first started doing research on A Confederacy of Dunces, I trudged through all of the Amazon reviews of the book, to see if anyone there had already put forward my ideas about Chaucer's influence on Toole. One of the approximately one thousand reviews argued passionately that Ignatius Reilly was clearly a person with a mental illness. To that reviewer, the book was an extended exercise in laughing at a mentally ill person and his misfortune. Shame on all of us. This perspective is similar to Patteson and Sauret's third Ignatius, as I described in my blog entry of September 1, 2020.
