Two days ago I compared Brian Leiter’s philosophy and law school rankings in a post (“Before You Consult the 2014 Philosophical Gourmet Report, Consider Leiter’s Words: “Reputation tends to be yesterday’s news”). I referred to comments Leiter made in 2010 about the relative value of reputational surveys and scholarly impact studies. Yesterday Leiter declared on his blog that I had taken his quotation out of context and misrepresented his views. This is what Leiter says in his post:
The “vitriolic criticsms” [sic] are understandable and also unrepresentative–obviously the critics have tremendous incentives to be a salient presence on social media, given the huge influence the PGR actually has in the real world. As I’ve noted before, it’s important not to be misled by the volume and persistence of the critics–pay attention to who they are, where they teach, where they earned their PhD, this will usually tell you more about what’s really going on. Not all of them have self-serving motives*, to be sure–some just have no judgment (vide Velleman). . . .
*Some are also pathologically dishonest, and are getting increasingly desperate now that the PGR is out. The most amusing is the former SPEP Advocacy Committee member who purports to quote me saying, “Reputation tends to be yesterday’s news–what happened 25 years ago,” without noting that I was discussing the awful U.S. News surveys of law schools, which are random surveys (not suveys [sic] of experts) and which provide the respondents with no information at all–of course, those kinds of surveys are yesterday’s news. The other big difference between academic law and academic philosophy is that in the former there is far less consensus on scholarly paradigms than in the latter.
I made two basic claims in my original post: first, that Leiter gave up reputational surveys for impact studies in his law rankings, and, second, that he thinks that impact studies are superior to reputational surveys. This second claim was based on the decision he made to give up reputational surveys in favor of impact studies as well as on his comments about the superiority of impact studies over reputational surveys.
The problem that Leiter faces in trying to confine his comments to criticism of the U.S. News reputational survey is that he doesn’t qualify his claim in this manner in the article I cited. Here once again are the remarks that I quoted in my post.
Most of the law schools in the top twenty are not surprising. But Leiter and Sisk agree that the study is a good indicator for future reputation.
“[Scholarly] Impact tells you things that reputation doesn’t,” Leiter said. “Reputation tends to be yesterday’s news–what happened 25 years ago. I think [this study] is useful for students who care about the academic experience” [brackets in original].
This quotation states that “[Scholarly] Impact tells you things that reputation doesn’t.” There is no qualification here: he doesn’t except his own expert-driven reputational surveys. In addition, he drops this portion of the quotation in his criticism of my post and only quotes the second sentence. “Reputation tends to be yesterday’s news–what happened 25 years ago.” This allows him to suggest that he is only talking about the U.S. News survey, or at least to make that reading more plausible. Restore the rest of the quotation and my point is all the stronger.
But there’s another difficulty with Leiter’s story:
The last reputational survey Leiter did for his law school rankings was in 2003. As I point out in my original post, the reputational rankings were later replaced by impact studies based on citations. (Leiter has also relied on lists of inductees to the American Academy of Arts and Sciences to rank faculty quality in the same period.) The shift to impact studies started in 2005, and is beyond dispute. What remains a mystery is why the shift, especially since Leiter was quite happy with the reputational survey approach for his law school rankings, and championed it in the same terms as he now describes the methods of the PGR–look at all of the fabulous people participating, etc. Here is Leiter in his “Introduction” to the 2003-2004 reputational survey results for his law school rankings:
Since high-quality survey data may ultimately be more informative than “objective” measures, it is my intent, for now, to rely on this data (emphasis added).
Further down the page he says this:
The quality of evaluators in this survey is unparalleled: it includes the President and President-elect of the Association of American Law Schools; a dozen members of the American Academy of Arts & Sciences, the nation’s most prestigious learned society; dozens of the most frequently cited legal scholars in numerous fields; and leading figures, junior and senior, in corporate law, criminal law, health law, constitutional law, jurisprudence, international law, comparative law, legal history, feminist legal theory, and many other fields.
Sounds like he has a pretty good thing going. But then the reputational surveys stop, and Leiter begins using impact studies based on citations. So the other difficulty that his story faces is that he abandoned reputational surveys for impact studies in his law school rankings. Why would he do this if he didn’t believe that impact studies were superior to reputational surveys for law school rankings?** The quotation I cited is what one would expect to hear after Leiter decided to switch.
Finally, a word about the last line in his criticism of my post. Leiter claims that there is “far less consensus on scholarly paradigms” in law than there is in academic philosophy.*** He appears to hope that people will believe that this is the reason that he stopped doing reputational surveys in law while continuing to use them for philosophy. The problem is, it’s not believable. Philosophy is certainly not enjoying a greater consensus about “scholarly paradigms” than law schools(!), and asserting that it does is completely unconvincing as a reason for assessing the fields differently. At best it’s a piece of wishful thinking on Leiter’s part, with the PGR itself something of a fantasy.
**As I noted in my post, the U.S. News’s rankings are only mentioned at the start of the article in order to contrast them with what Sisk and his colleagues were doing, with a very brief mention that part of what U.S. News does is reputational. The article then moves on, with Leiter’s words coming near the very end. I should add that in the middle of the piece there is another quotation from Leiter regarding how “Scholarly impact is a measure of the intellectual quality of the faculty…” Nothing more is said about U.S. News’s rankings.
*** UPDATE: 12:00 PM, December 10. I inadvertently switched law and academic philosophy in this sentence when I first published the post early this morning. It’s been corrected. Thanks to John Protevi for catching it.