In my new paper, just published as a book chapter in Theology and Geometry: Essays on John Kennedy Toole’s A Confederacy of Dunces, I briefly lay out a theory of humor (for a summary, see my blog entry Theory of Humor, part 9). I end the discussion by noting the possible evolutionary function of the social dimension of humor. "Here is what I say in the text: "This social aspect of humor suggests a path for the evolution of laughter and humor as a mechanism for the maintenance of pro-social behavior in our species." Here is the endnote (enhanced with full citations and paragraph breaks):
Human laughter and humor evolved from the pant-hoots that other primates utter during youthful mock aggression. Some theories about this evolution are part of the effort to explain why humans can be altruistic despite our selfish genes. A proposal by Gervais and Wilson comes from “multilevel selection theory.” In that theory, human groups use low-cost social mechanisms for promoting altruistic behavior, and humor is one such low-cost mechanism. See Matthew Gervais and David Sloan Wilson, “The Evolution and Functions of Laughter and Humor: A Synthetic Approach,” The Quarterly Review of Biology 80 (2005): 395-430.Not only do I argue for the pro-social evolutionary function of humor, but I show how even in the early 18th century that altruistic aspect of humor was recognized by Hutcheson in rebuttal to Thomas Hobbes. This observation comes under the category of evolutionary literary criticism or evolutionary narratology.Another evolutionary theory is that of the altruistic punisher. An altruistic punisher is a group member who, through personal cost, enforces group cooperation. Other group members then must support and compensate this punisher in order for the dynamic to be evolutionarily stable. William Flesch describes a theory of tragic heroes as altruistic punishers in Comeuppance: Costly Signaling, Altruistic Punishment, and Other Biological Components of Fiction (Cambridge, MA: Harvard University Press, 2007). In comedy, both the group member who uses humor to lower the status of other group members and the comic role of the trickster can be seen as altruistic punishers, and the laughter provoked by their efforts can be seen as second order support for them.
These two theories, multilevel selection theory and altruistic punishment, are not incompatible.
The question of altruism was also part of the early discussions of humor; Hutcheson criticized Hobbes for basing his worldview on selfishness, “Now natural affections and kind instincts are banished from philosophy,” in Francis Hutcheson, Reflections upon Laughter (Glasgow: Baxter, 1750, but originally published serially in 1726), p. 7.
Hurley, Dennett, and Adams use an evolutionary framework for their theory, but they neglect the social function of humor to their detriment. See Matthew M. Hurley, Daniel C. Dennett, and Reginald B. Adams. Inside Jokes: Using Humor to Reverse Engineer the Mind. Cambridge, MA: MIT Press, 2011.