Two months ago, I posted a blog entry called Toole's Dunces Chatbot, which was about my initial attempt to create a simple GPT chatbot in ChatGPT that could answer questions about John Kennedy Toole's A Confederacy of Dunces, with the training data coming from my own writings. I described how the chatbot, which I call Toole's Dunces Chatbot, despite being fed good answers in the training data, often responded to prompts with vague or incorrect answers. In short, it was often bullshitting me.
Last month, I posted a blog entry called Toolebot in NotebookLM, which recounted how I used the same training documents and the same battery of questions with Google's NotebookLM system and called the chatbot the Toolebot. Toolebot gave answers that were far superior to Toole's Dunces Chatbot. At the end of that blog entry, I mentioned that a colleague had suggested that I turn off the ChatGPT option for it to search the internet for answers. That didn't help. The answers to my questions were still bad.
In mid-October, I participated in a discussion with other faculty at my university about chatbots. There, I learned that I should try to improve the prompt instructions for my chatbot. I have learned only a tiny bit about what is called prompt engineering. When I set up Toole's Dunces Chatbot, I only answered a half-dozen questions asked by the configuration chatbot. Those answers stocked a small list of prompt instructions. One can create much more elaborate sets of instructions, and the resulting chatbot can give better answers. A colleague shared with me some prompt instructions he uses for his GPT chatbots, and I added some of them. They really did improve the output. I give some examples below.
Summary: Although improved instructions improved the GPT output, it is still inferior to the basic output from Toolebot without enhanced prompt instructions.
Details:
Here is the set of instructions that Toole's Dunces Chatbot had when I reported on the early tests I conducted in August:
Instructions: This GPT serves as a scholarly guide to the novel 'A Confederacy of Dunces' by John Kennedy Toole. It provides insightful information, interpretations, and discussions about the themes, characters, and historical context of the novel. It also offers analyses and answers questions related to the text, helping users understand and appreciate the complexities and humor of the work. The GPT should always focus on accuracy, depth, and literary understanding, while maintaining an approachable and engaging tone. Avoid engaging with or endorsing the theory that John Kennedy Toole did not write the novel. Responses should be formal, reflecting the scholarly nature of the content.Here is the additional set of instructions that I added in October:
Always search your knowledge before completing any prompt and use only that knowledge in your completion. Check EVERY document in your knowledge before you answer. MOST IMPORTANT: Only use information in your knowledge; Never use any information not contained in your knowledge. Do not use general information or knowledge. Double check your answer to be sure it is accurate. Do not add extra information to an answer.
A Sample Prompt and the Answers in Different Chatbots: Three Ignatiuses
Here is a section of a note from my "Comic Mechanisms" paper, which I then quoted in this blog in September of 2020. This text explains the three Ignatiuses of Patteson and Sauret.
Patteson and Sauret perceptively argue that Ignatius’s situation is more complex that a simple contrast. To them, there are three versions of Ignatius in the book. First, he sees himself as a crusader against corruption; second, the other characters see him as a selfish buffoon; however, the third Ignatius has retreated into a world of illusion after a painfully awkward childhood in order to avoid being humiliated by rejection and alienation. Ignatius’s outrageously childish behavior insulates the reader from feeling too much pity for him, but this third Ignatius gives the reader some sympathy for him, even as the reader laughs at his humiliations.The Prompt: "According to Patteson and Sauret, who are the three Ignatiuses?"
Here is the answer from Toole's Dunces Chatbot when it had minimal prompt instructions and the ability to search the Internet.
According to Patteson and Sauret, the three "Ignatiuses" are different aspects or roles that Ignatius J. Reilly embodies throughout A Confederacy of Dunces. These are:Comment: This version of Toole's Dunces Chatbot got all three wrong.Ignatius as a Picaro: Ignatius is seen as a picaresque character, a rogue or anti-hero who navigates through a corrupt society.
Ignatius as a Holy Fool: He is also depicted as a "holy fool," someone who is ridiculed by society but possesses a deeper, often misunderstood wisdom.
Ignatius as a Scapegoat: Finally, Ignatius functions as a scapegoat, a character upon whom the other characters project their frustrations and blame for their own shortcomings.
Here is the answer from Toole's Dunces Chatbot when it had minimal prompt instructions, but with the ability to search the Internet turned off:
Ignatius as the Academic: He is depicted as a parody of an intellectual who spouts philosophical and medieval ideals but is incapable of meaningful academic contribution. His pretensions reflect the failure of the intellectual life when disconnected from reality.Comment: Once again, this version of Toole's Dunces Chatbot got all three wrong.Ignatius as the Carnival Figure: Drawing on the concept of Carnival, Ignatius represents chaos and inversion of social norms, much like a lord of misrule. He disrupts social order, causing mayhem, but also allowing for renewal through disorder.
Ignatius as the Picaro: He exhibits characteristics of the picaresque hero a rogue living on the margins of society, whose episodic misadventures expose the hypocrisies and absurdities of the world around him.
Here is the answer from Toole's Dunces Chatbot with enhanced prompt instructions and with the ability to search the Internet turned off:
The Crusader: This is how Ignatius sees himself--a champion against the corruption of the modern world, viewing his actions as part of a grandiose intellectual and moral crusade.Comment: Finally, ChatGPT gets the correct answer. Prompt engineering is effective for ChatGPT.The Buffoon: This is how other characters see him--a selfish and absurd figure whose behavior and schemes are laughable and ridiculous.
The Tragic Figure: This is the hidden Ignatius who has retreated into a world of illusion after a painful and humiliating childhood. His eccentric and childish behavior is a defense mechanism against the alienation and rejection he has experienced, making him a figure of both comedy and sympathy.
Here is the answer from NotebookLM's Toolebot in its out-of-the-box configuration:
Patteson and Sauret, in their analysis of John Kennedy Toole's A Confederacy of Dunces, present a nuanced perspective on the character of Ignatius Reilly by proposing that there are three distinct versions of Ignatius present within the novel. These versions are distinguished by how Ignatius perceives himself versus how he is perceived by others, adding layers to his characterization beyond a simplistic interpretation. CITES 2020/9Comment: NotebookLM hits it out of the park on the first try, and it cites the correct blog entry. It then reflects on the meaning of the three Ignatiuses and cites another blog entry to that effect. It is still the best. This example shows GPT eventually answering the prompt correctly. In other prompts from the battery of test questions, ChatGPT still did not get correct answers.The Crusader: This is the Ignatius as he sees himself: a righteous warrior battling against the perceived corruption and vulgarity of modern society. He fancies himself an intellectual, a defender of "theology and geometry," and a champion of a bygone era.
The Pompous Fool: The second Ignatius is the one presented through the narrator's descriptions and the perspectives of the other characters. They see him as a selfish, lazy, and ridiculous buffoon, blind to his own hypocrisy and shortcomings.
The Vulnerable Soul: The third Ignatius adds a layer of complexity and evokes a degree of sympathy from the reader. This Ignatius is a product of a difficult and awkward past, and his outlandish behavior serves as a defense mechanism to shield himself from further humiliation and rejection.
According to the sources, the interplay between these three Ignatiuses fuels the humor of the novel. While the reader laughs at the antics of the pompous fool, the presence of the vulnerable soul adds a tinge of tragedy and encourages a more compassionate understanding of his character. CITES BLOG 2021/2 It is the third Ignatius, the one who retreats into a world of illusions to cope with a painful reality, that allows the reader to recognize the pathetic and perhaps even tragic elements beneath the surface of the comedic farce.
A further example: Because I have in the earlier blog entries used the prompt about damning characters, I will mention that as a good example where ChatGPT still fails. Even with the improved prompt instructions, Toole's Dunces Chatbot still claims that Ignatius regularly damns the other characters. That is simply not true. ChatGPT conflates insulting someone and actually saying, "Damn you." Because it does not differentiate those two situations, it does not realize that Ignatius, despite his abusive behavior, never damns anyone. The Toolebot aced the question about damning.