“Ideological blindness at the Supreme Court biases the justices in favor of big business.” “Poor organizational design is causing bad lawyering by the Solicitor General before the Supreme Court.” These are just two of the indictments at the heart of a new book demonstrating an unfortunate trend in the social sciences. Social scientists, including economists, who apply quantitative methodologies to assess qualitative disciplines are not producing findings nearly as conclusive as they think. Nearly all of their “groundbreaking” studies and journal articles fall victim to the same mistake: they impose a false sense of order on subjects that defy neat sorting.
Justice Oliver Wendell Holmes once quipped “a page of history is worth a volume of logic.” In evaluating Clifford Winston’s, David Burk’s, and Jia Yan’s Trouble at the Bar: An Economics Perspective on the Legal Profession and the Case for Fundamental Reform, we might adapt Holmes’s dictum in this way: “a short conversation with a practicing advocate is worth a volume of regression analyses.”
Trouble at the Bar surveys, from the perspective of one senior Fellow at the Brookings Institution and two academic economists, an extensive variety of internal issues that the authors believe plague the practice of law in modern America. The authors view these issues through the lens of economics, attempting to reveal insights from quantitative assessments that others, including practicing lawyers busy advancing their careers, have been unable to measure and remedy.
While Trouble at the Bar’s three authors, none of whom have practiced law or hold a juris doctorate, helpfully aggregate data and perform regression analyses on LSAT scores, law school rankings, and attorneys’ earnings premiums, these types of analyses act more like hammers in search of nails when applied to wide-scale analyses of actual legal questions and specific personalities. They are improper tools for analyzing, in particular, the work of the Office of U.S. Solicitor General and the judicial reasoning of current and recent justices of the U.S. Supreme Court. The authors should have thought twice about their methods before applying them to the members of the legal profession they criticize. The book is a useful reminder therefore of the limits of economic analysis on the practice of law.
The Solicitor’s General Office
In trying to analyze inefficiencies resulting from resource constraints among government lawyers, the authors perform a case study on the work of the Office of U.S. Solicitor General. Unfortunately, such a unique office is ill-suited for their analysis. The first problem concerns a formula that the authors created to assess measuring case outcomes at the Supreme Court for those who work in the Solicitor General’s office.
The formula reads: Pr(W) = f(C,I,L),
where Pr(W) is the probability that a SCOTUS case ruling favors the government instead of a private entity; C encapsulates the legal issues and ambiguities that underlie the case in question; I represents the preferences and ideologies of the justices hearing the legal dispute; and L captures differences in the quality of the government and nongovernment legal representation, which we specify in terms of whether the government or the private entity is represented by a top Supreme Court advocate.
One need only be a law student to realize that the grouping constraints imposed by the authors on each of these variables are tenuous at best. How many layers of assumptions comprise the judgment that a ruling “favors the government” rather than “a private entity?” How does one compare the relative legal issues and ambiguities of one case versus another?
The authors tick off twelve extremely broad issue areas for examination, placing cases into certain substantive boxes as if the justices themselves considered cases as neatly fitting into these twelve areas such as federal power, First Amendment, taxation, and the helpful label “miscellaneous.” The quixotic quest to define the preferences and ideologies of individual justices will be addressed later in this review. But the method by which the authors measure the differences in the quality of legal representation serves as a microcosm for the errors that plague their analysis.
To assess whether resource constraints affect the work of employees of the Office of U.S. Solicitor General, the authors choose eleven “Top Supreme Court Advocates” who have worked on cases at the Supreme Court. The authors track these advocates’ years in the federal government and out of the federal government. You have likely heard of most of these advocates, including Paul Clement, Don Verilli, and yes even an eventual Supreme Court justice, John Roberts.
The authors ask whether those resource constraints constitute a governmental failure worth addressing. After running their evaluation through 527 split decisions at the Supreme Court involving those eleven advocates, the authors conclude “the chances of a government win are substantially reduced when its private rivals are represented by a top Supreme Court advocate. . . . In contrast, when those same lawyers bring their skills to represent the federal government, their effects on the probability of a government win are quantitatively negligible and statistically insignificant.”
The authors should realize the limits to the tools in their economist’s kit when criticizing certain aspects of the legal profession.
However, the authors do not prove much of anything. Their holding rests on some fatal conceits. First, the authors are working with small numbers of idiosyncratic personalities and varying circumstances. Of the eleven “Top Supreme Court Advocates,” the authors base the performance of more than half of the advocates on representative cases or a case from a single year of employment outside of the federal government. For example, the authors want us to believe that Stephen M. Shapiro’s performance on cases in 1980, while he worked in the federal government, tells us anything of note about the relative quality of lawyering he performed on cases he litigated in 2004. The authors warn that “unobserved variables” may affect outcomes while also admitting that the stage of an advocate’s career will affect whether he or she takes a difficult case in private practice. The breadth of this admission swallows the validity of the authors’ claims. If the authors purport that their findings have any consistency, these findings would not be so reliant on the idiosyncratic choices made by such a small sample of top advocates.
Another area of the authors’ concern is whether resource constraints in the Office of the U.S. Solicitor General affect the defense of the interests of the government. The authors are correct when they report that briefs the Office submits to the Supreme Court are more important to securing the interests of the government than the Office’s performance at oral argument. The authors blame substandard briefs at least in part on the organizational design of the Office as a team of under twenty-five attorneys working outside their areas of specialty. While this criticism has merit, the authors’ analysis suffers from another incorrect assumption: the Office’s organizational design greatly affects its performance.
There are several reasons to dispute the authors’ linking organizational design with poor performance. First, the prestige of working in the Office of Solicitor General has steadily been maintained over the past decades. One need only regard the sterling resumes in recent years of Bristow Fellows, recent law school graduates who devote one year of their promising careers working in the Office. The Office also maintains that prestige in times of filing relatively high numbers of cert. petitions and in times when it files relatively fewer number of cert. petitions. Adam D. Chandler has observed, “[T]he extent to which that reputation is protected by seeking cert. narrowly has been exaggerated. No one suggests that the Solicitor General was any less credible in the 1980s when he was filing three times as many petitions as he is today.”
Moreover, while the Office is the most important litigant before the Supreme Court due to its singular role as representing the views of the federal government and as a repeat litigant, the Office’s briefs before the Supreme Court less and less convey novel arguments. Particularly true in cases at the Supreme Court in which the Office is not the sole litigant, a brief that the Office submits in a case is now one among a growing number of amicus briefs submitted to the Court. Less and less is the Office of Solicitor General singularly responsible for making novel arguments before the justices. Therefore, if any governmental failure exists, it rests more with the justices of the Supreme Court for not responding to arguments made in the other amicus briefs they receive.
The authors would have done an enormous service, and improved the quality of their research, if they had approached the members of the “Top Supreme Court Advocate” group to ask if potentially limited resources affected their work in the Office. Far more conclusive would have been a series of interviews with those advocates and whether the data the authors collected confirmed or contrasted with the statements of those advocates.
The Ideology of Supreme Court Justices
The tenuous assumptions relied upon by the authors in analyzing the work of the Office of U.S. Solicitor General pale in comparison to the untethered assumptions the authors make in their chapter on the ideology of Supreme Court justices. In short, the authors present not only a narrow analysis based on what they label “pro-business” outcomes, but they also draw from their analysis a sweeping, unfounded indictment of judicial independence.
Close analysis of the Roberts Court’s pro-business leanings is, according to the authors, a window into how the Court’s decision-making is based not solely on legal reasoning, but also on how the justices’ political ideology affects the justices’ legal reasoning. The authors draw inspiration for this approach from a notable law review article on the Roberts Court and its rulings in business cases. “We find that justices’ ideologies have a statistically and quantitatively important influence on their rulings on business cases,” the authors assert.
The problems with attributing, even if only in part, deterministic force to the justices’ political ideology on the outcomes of cases are myriad. First, the vast majority of cases to reach the Supreme Court are “close calls” that defy simple notions of having litigants classified as “pro-business” versus “anti-business.” For example, in the recent case of Google v. Oracle, which company represented the “pro-business” interest and which company represented the “anti-business” interest? Additionally, legal arguments defy ever-shifting political groupings part of the popular imagination. While Republican-appointed justices may have demonstrated a pattern of supporting some “pro-business” litigants in the past, decisions such as Justice Clarence Thomas’s concurrence in Biden v. Knight First Amendment Institute, in which the justice expressed an openness to revisiting the constitutionality of protections for internet companies under Section 230 of the Communications Decency Act, muddy the distinctions for what constitutes pro- versus anti-business litigants.
When the authors review the “box scores” of how certain cases were decided, looking for patterns between political ideology and judicial ideology, they track seemingly everything but the most important criteria: the actual judicial reasoning in those decisions. But is such an impartial assessment of judicial reasoning possible or even helpful? Any cross-categorization of judicial reasoning across opinions by different justices obscures enormous variables influencing how the justices consider an outcome just. The econometric model the authors propose for understanding the outcome of cases is more dizzying than coherent, though this reviewer admits he is not its intended audience. Nevertheless, what was clear was that the authors neglected to view these cases through the eyes of the justices who, to a person, profess to approach each case uniquely, not as reflective of political and judicial ideology.
Thus, their analysis of the justices reads as a ham-fisted attempt by economists to impose a kind of order on subjects, judicial opinions, that defy the simplistic categorization that the authors impose. As legal commentator Edward Whelan observed when critiquing a separate study of judicial voting habits co-authored by legal scholar Lee Epstein whom the authors of Trouble at the Bar rely upon, “I am open to the possibility that statistical analysis of voting patterns might discover apparent anomalies that suggest avenues for further exploration. But that further exploration, if it is going to be fruitful, will have to involve the sometimes difficult and often contestable work of legal reasoning—work that is beyond the bounds that modern political scientists have imposed on themselves.”
One can demand the same of the economist authors of Trouble at the Bar as Whelan demands of political scientists. The authors should realize the limits to the tools in their economist’s kit when criticizing certain aspects of the legal profession. Those in the legal profession they place in their crosshairs would treat their analyses more seriously if the authors admitted that the economist’s tools can analyze only so much.
This article originally appeared on Law and Liberty and can be read here.
