Saturday, July 1, 2023

Comparison Study of AI Systems for Library Research and Scholarship (and Confederacy of Dunces)

Back on April 1, I posted a blog entry called The Wisdom of ChatGPT. I prompted ChatGPT with a question about John Kennedy Toole's novel A Confederacy of Dunces and the philosopher Marsilio Ficino, and then I shared the resulting essay. It was obvious that ChatGPT could write flowing English prose, and it made valid but vague points and seemed to be able to express ideas, not just produce text. Still, ChatGPT did not "know" information that could be found in my paper on the topic published in the journal Renascence, information that was readily available on the Internet. My own papers may not have been in the universe of documents from which ChatGPT learned English. Either that, or it lacks the capacity to actually store knowledge.

Since then, I read an article about several AI-assisted systems that either write text or conduct library research on a user's behalf. On June 20, I decided to run my query about Confederacy and Ficino in several of these systems. Below are my assessments of the results. I will begin with the systems that were easy to evaluate.

Query: In his novel A Confederacy of Dunces, did John Kennedy Toole use the ideas of Marsilio Ficino?

System: SciSpace; Time Query Executed: June 20, 2023, 3:10 p.m. U.S. Central Time; URL: https://typeset.io/

Review: SciSpace did not write an essay but returned citations to articles, as a search engine would. It returned my own article on Confederacy and Ficino as the number one result, and another text I had written which mentioned my ideas as the number two result. For doing library research, it performed well with this prompt. If this system does as well in other domains, it could genuinely help scholars.

System: Elicit; Time Query Executed: June 20, 2023, 11 a.m. U.S. Central Time; URL: https://elicit.org/

Review: Elicit wrote a summary paragraph and returned search results. The paragraph was not bad but saw no connection between Confederacy and Ficino. Unless a system has digested my own articles, it is likely to say that there is no connection, so I don’t fault this system's algorithm, just its knowledgebase. The citations were a mix of articles about Confederacy and articles about Ficino. The articles were not the best, but they were real academic articles and not completely off topic. Okay, but not great.

System: Bing AI; Time Query Executed: June 20, 2023, 3:26 p.m. U.S. Central Time; URL: https://www.bing.com/?/ai

Review: Bing returned search results. The results were not good. The first result was the Confederacy of Dunces Wikipedia page, then an article from the New Yorker on the novel, then the Goodreads page for the novel. Bing does not claim to be a search system for scholarship, but it is a comprehensive search engine. This was very weak. Still, it knew who John Kennedy Toole was.

System: Consensus; Time Query Executed: June 20, 2023, 3:05 p.m. U.S. Central Time; URL: https://consensus.app/

Review: Consensus returned search results without writing an essay or summary. The results were terrible. Results included articles on George Kennedy, Supreme Court Justice Anthony Kennedy, President John F. Kennedy, and an essay about the letters of Marsilio Ficino. Yikes.

System: Scite.AI; URL: https://scite.ai/assistant

Review: This system would not give me a free trial unless I put in my credit card information. No thanks.

Here is the hard one

System: Google Bard; Time Query Executed: June 20, 2023, 3:22 p.m. US Central time; URL: https://bard.google.com/

Review: Bard did the best job of writing an essay. It wrote a short essay on the topic with no citations. I struggle to assess its performance because it might be genuinely good, or it might not. It might know more about Ficino than I do, or it might just be an excellent deep fake. I wrestle with whether Bard came up short or I came up short.

Here is the Bard essay

Yes, there is some evidence to suggest that John Kennedy Toole used the ideas of Marsilio Ficino in his novel A Confederacy of Dunces. In particular, the character of Ignatius J. Reilly has been interpreted as a representation of the melancholic genius, a figure that was theorized by Ficino.

Ficino was an Italian Renaissance philosopher who believed that there were three types of souls: the rational soul, the spirited soul, and the appetitive soul. He associated the rational soul with the planet Mercury, the spirited soul with the planet Mars, and the appetitive soul with the planet Saturn.

Ficino also believed that each person was born with a dominant soul, and that this soul determined their personality and temperament. People with a dominant rational soul were rational and intellectual, while people with a dominant spirited soul were passionate and energetic. People with a dominant appetitive soul were materialistic and sensual.

The character of Ignatius J. Reilly can be seen as a representation of the melancholic genius. Ignatius is intelligent and perceptive, but he is also deeply pessimistic and withdrawn. He is constantly complaining about the world around him, and he seems to have little interest in engaging with it.

Ignatius's melancholy can be seen as a manifestation of his dominant saturnine soul. Saturn was associated with melancholy in ancient astrology, and Ficino believed that the melancholic genius was a child of Saturn. Melancholic geniuses were seen as being highly creative and intelligent, but they were also often troubled and isolated.

Of course, it is impossible to say for sure whether Toole was consciously drawing on Ficino's ideas when he created the character of Ignatius J. Reilly. However, the similarities between Ignatius and the melancholic genius are striking, and it seems likely that Toole was at least familiar with Ficino's work.

Discussion

In my own scholarship, I argue that Ignatius was a parody of Ficino's melancholic genius. I also argue that Ficino sees the genius as a child of Saturn. However, I was unaware that the three souls were important to Ficino and that the appetitive soul is necessarily associated with Saturn. The question is: who knows his Ficino better, me or Bard?

Most of what I know about Ficino comes from the book The Philosophy of Marsilio Ficino by Paul Oskar Kristeller. Reading this essay by Bard made me go back to Kristeller and see if he discussed the rational soul, the spirited soul, and the appetitive soul, as Bard claims. Although Plato discussed those three parts of the soul, and Ficino was a thorough translator of Plato, Kristeller dismisses the three souls as an aspect of Ficino's philosophical system. Kristeller states, "Though Ficino occasionally quotes Plato's distinction between the rational, the courageous, and the appetitive parts of the Soul, he does not assign any systematic significance to it" (367).

The question remains, is Kristeller right, or does Bard know more about Ficino than Kristeller does? If Bard is wrong, then did Bard just grab the facts about the Platonic division of the soul and mash them together with Ficino's melancholic genius and astrological theories?

I am inclined to trust Kristeller, especially since he devoted years to Ficino and was the first scholar to turn Ficino's writings into a systematic philosophy. I come back to the fact that Ficino saw two possible types of children of Saturn, one who was beastly and one who was divine. To me, Toole's joke is that Ignatius thinks he is a divine child of Saturn but is in reality a beastly child of Saturn. In the end, I think that Bard took several ideas about Ficino and Platonic philosophy and glued them together, a very good deep fake of genuine scholarship.

Nevertheless, Bard seems to know things. It seems to possess an internal set of ideas. ChatGPT seemed to know a bit, but it was otherwise vague and non-committal. Bard, by contrast, had definite ideas, some of which it may have gotten by digesting my own writings, but maybe not. Not only did Bard know who John Kennedy Toole was, and what Confederacy of Dunces was, but it also knew who Ficino was. Even if Ficino did not assign systematic significance to Plato's three souls, he really did talk about them. It is even possible that Bard knows enough about Ficino that it could challenge Kristeller's interpretation of Ficino's work (and therefore my interpretation of Toole).

In short, Bard is so good, it made an expert on a topic go back to his books to reassess the issue. No other AI system in this line-up did THAT.