Testing Limits

THE BIG TEST:The Secret History of the

406 pages. $27.00.

SAT : meritocracy ::

A. farrier : horseshoe

B. heredity : aristocracy

C. portabello : mushroom

D. meter : length

E. modems : Internet

Let’s hold off on the analogy (often an excellent strategy on hard SAT questions) to consider some salient facts about the SAT’s history unfortunately missing from The Big Test. In this ostensibly comprehensive history, Lemann elides any substantive discussion of (1) how the test has changed across the years (in this decade alone, changes include the elimination of antonym questions; the addition of free-response math questions; the elimination of the grammar section; the addition of two shorter, more-focused sections; the “re-centering’ of the scoring calculus; and the popularization of a secondary test with graded essays); and (2) just as importantly, the astonishingly large percentage of the test that has barely changed at all since the test was developed in the years before World War I. Lemann passes up the opportunity to explain some of the frightening figures that he randomly sprinkles throughout the book (for example, the correlation of SAT scores to first year college grades is no more accurate than the correlation to parents’ household income). He declines to elaborate on the epidemic problem of cheating on the SAT, or on the arbitrary and malevolent way in which the College Board handles cases of suspected cheating. He glosses over the fact that after more than six decades as the “Scholastic Aptitude Test,” in 1994 the SAT first adjusted its acronym to the “Scholastic Assessment Test’ — an admission that the exam failed as a test of innate “Intelligence Quotient’ (I.Q.) and instead measured learnable skills; and then again changed it to stand for nothing but its own famous acronym, “SAT’ — perhaps an admission that the test provides no categorizable or useful measurement, at all.

If my bias isn’t already clear, let me disclose it. In high school, I was subjected to the tedious poking and prodding of standardized tests now routine for students who aspire to attend competitive universities. I took the SAT twice (not counting having taken it in seventh grade for a scholarship competition) and its primary competitor, the ACT; before that I took the PSAT twice; across the years I took more than a handful of Advanced Placement (AP) exams; as a senior I took a series of Achievement Tests (now called the SAT-IIs), including the much-heralded Writing Exam, which includes essays. Since then, I have taken the Graduate Record Exam (GRE) and the GRE Subject Test in English Literature, and the Law School Admissions Test (LSAT). With the exception of the ACT, every one of those tests originates from the Educational Testing Service (E.T.S.). Like most who have survived the gamut of these tests, I resent their existence, their importance, and their makers. More immediately, I have spent the past five years teaching test preparation, part time, for The Princeton Review, a private company that specializes in this now booming industry. Lemann is pitch-perfect on The Princeton Review’s ethos, when he describes the preparation courses as “taking the position that SATs [are] pernicious, meaningless bullshit foisted upon America’s youth by a greedy corporation.”

I fully expected a thoroughly researched, well documented, engagingly written defenestration of that greedy corporation — the Educational Testing Service — from The Big Test. It never occurred to me that The Big Test would stray from the journalistic fairness that has characterized Lemann’s earlier work, from his prize-winning previous book, The Promised Land, to his lengthy articles in the Atlantic Monthly and the New Yorker. Simply put, I retain an unshakable faith that a thoroughly researched, well documented, engagingly written book on the SAT could yield nothing short of a full-scale indictment of the test and its mythical status in our society.

Yet despite the impression conveyed by its title, Lemann’s book is not really about the SAT. The subtitle, with its tight focus on “the American meritocracy,” is a much truer indicator of what’s to be found in these pages. Ultimately, I found The Big Test to be a thoroughly researched, well documented, engagingly written treatise on the evolution of meritocracy as an idea and an American ideal. This is most certainly a fascinating and deserving topic, worthy of serious, lengthy examination. The problem with this lengthy examination is that it applies a journalistic methodology to what is finally a philosophical question: What are the moral consequences of the ways in which American meritocracy has evolved? The book outlines sweeping questions with abstract parameters, but seeks answers in focused anecdotes. Lemann’s profiles are vivid, engaging, and often illuminating, but even collectively, they fall far short of comprising a compelling logical framework on which to build any sort of answer to the broad questions his book poses.

Let’s return briefly to the analogy at the top of this review. Generally speaking, the best way to approach an SAT analogy is to link the two words from the question in a simple sentence. Yet, the relationship between “SAT’ and “meritocracy” is not instantly clear. In this case, then, the best strategy is to make sentences out of the answer choices and attempt to fit the question words into those sentences. Looking at answer choice A, we can construct the sentence a farrier makes horseshoes. The SAT does not, however, create a meritocracy, even if it does contribute. Answer choice B provides the sentence heredity gets one into the aristocracy. The SAT may be the ticket to the aristocracy, but that’s a bit iffy. A portabello is a type of mushroom, but that relationship clearly does not apply, while a meter is a measure of length, and that too fails to meet our needs. The relationship in the last answer choice is best explained by modems provide access to the Internet. Does the SAT provide access to the meritocracy? At the very least, it is more accurate to say that the SAT provides access (answer choice E) than that it is a guaranteed ticket (answer choice B).

But while high SAT scores are not a guaranteed ticket to elite colleges and membership in the intelligentsia (the ideal of the meritocracy?), low SAT scores are more and more denying people access to those ends. Indeed, when Lemann’s narrative begins, Henry Chauncey (who later became the first president of E.T.S.) is an assistant dean at Harvard. He got there not in the way we would expect today — an overachieving high school career leading to a stellar collegiate experience at an elite undergraduate institution, then a top-notch graduate school, followed by a prize-winning publication earning tenure and a position in the Harvard hierarchy — but after an academically undistinguished time at Groton (the most connected prep school of the day) followed by a similarly unremarkable undergraduate career at Harvard. Harvard was then (and then was only sixty years ago) not an exclusive enclave for the erudite, but a prestigious playground for the elite’s children to come of age. Academic success was not integral to Harvard admissions, and assistant deanships were reserved for those without ready-made employment options upon their graduations. Henry Chauncey’s assistant deanship was the result of coming from a not-so-well-to-do family at the fringe of the upper class, showing athletic prowess at Groton and Harvard, and evidencing the noblesse oblige expected of good Christian elites. Then, Harvard Yard was reserved for men of Chauncey’s ilk; today, there would be no place for him there.

That this shift is due in large part to Chauncey’s own success in selling the SAT’s merits to colleges is undoubtedly ironic. But Chauncey’s test simply isn’t as good as he thought it was. Scores on the SAT are striated along race, gender, and, above all, class lines. (Who could have predicted that a test written at the beginning of the century by a group of white, Protestant, New England Brahmin men, and largely unchanged since, would fail to treat blacks, Hispanics, women, and the poor of all races as well as it treats those most like its creators?) As a result, numerically-obsessed college admissions offices tend to favor white, male, upper-class applicants. This state of affairs led to affirmative action programs — more prevalent in higher education admissions than in any other sphere of American society — to redress unequal distribution of opportunity. It was Henry Chauncey, Jr. (Sam to his friends) who smoothed the way for the installation of affirmative action programs at Yale.

The ostensible meritocracy created in an America where stellar SAT scores (judged within an affirmative action context) led to elite schooling and thus to success is the story of the middle third of The Big Test. This is the least cohesive section of the book, and the most frustrating. As soon as Lemann builds the narrative to the point that E.T.S. and Henry Chauncey are off the ground, he switches venues and gears. Yale takes Harvard’s place as the generic stand-in for elite higher education, and Sam Chauncey replaces his father as the primary character in the narrative. The other major figures we meet in this section — including Bill Lan Lee, Molly Munger, and Don Nakanishi, early beneficiaries of affirmative action all — have no apparent relevance to the larger narrative, and each of their stories is tinged with unpleasantly Horatio Algeresque undertones. This “American Dream’ undercurrent conflicts with the prevailing sense that the book is an expose of something unsavory and undercover. The section ends with an all-too-brief return to the E.T.S. story circa the early eighties, including fluffy discussions of the burgeoning test-prep market and a dismissive summary of the 1980 Allan Nairn and Ralph Nader report on E.T.S. (The Reign of E.T.S.: The Corporation That Makes Up Minds). Simply put, Nairn and Nader produced a powerful book despite E.T.S.; had they received the kind of access Lemann had, the result would have been far more thorough, far more revealing, and far more damning than The Big Test.

If we return for a final look at our ana-logy, we will find that it is apt for another reason, less obvious than the one we identified earlier. For the truth goes beyond the fact that the SAT provides access to the meritocracy. More than an instrument of the meritocracy, the SAT is simultaneously the primary limitation of the meritocracy. Modems were the instruments that created the Internet by allowing easy public access to it, but so too they are the Internet’s primary limitations, retarding the spread of technological advantages that would simply bog down and overwhelm commonly used modems. Similarly, the SAT fueled the creation of the meritocracy, but now limits its creation; the biases built into the test, along with the mythological power the test has in our society, stand as significant obstacles to creating a true, functional, equal opportunity, meritocratic system.

Lemann never actually comes out and says this in so many words. Instead, he devotes the final third of The Big Test to an in-depth examination of the political struggles over affirmative action in California. He builds the end of the book around the battle over Proposition 209 — the 1996 ballot initiative to abolish affirmative action in California — which unites many of the disparate threads from the middle of the book. Molly Munger, Bill Lan Lee, and Don Nakanishi, as well as new characters like Jerry Karabel and Glynn Custred, all have roles to play in the Prop 209 fight. Lemann takes no position on Prop 209, although he lavishes much more attention on its opponents than its supporters. In part, he does this because those opponents are the standard-bearers of the meritocracy as he has defined it. But, again, he leads the reader to root for these people, which complicates his conclusions, in which he yanks the rug out from under the entire meritocracy.

He is able to get from his support for Prop 209’s opponents to his opposition to the meritocracy as we know it because of Molly Munger. Munger, who was a trailblazing woman at Harvard Law School, in Los Angeles’ culture of corporate lawyers, and eventually at the N.A.A.C.P. Legal Defense Fund (where she spearheaded the campaign against 209), left lawyering after 209’s passage to start the non-profit Advancement Project. Lemann is vague on the Advancement Project’s goals, but he uses her disillusion with affirmative action to reach his conclusion: “What really, as opposed to rhetorically, transfixed late-twentieth-century America was the precise calibration of a systematic national reward system, which was what the testing and education regime had become over half a century. But that was secondary and morally unimportant, especially if the people who got the best rewards weren’t going to concern themselves with the good of the whole society.” This sentiment, though artfully expressed, is neither novel nor particularly deep. Yet, it takes Lemann almost 350 pages to get there — and then he leaps to some fairly rash conclusions.

Lemann’s Afterword is a screed to throw the baby out with the bathwater. He wants a national curriculum for public schools, enforced by a national achievement test (as opposed to an aptitude test), and leading to a 100 percent publicly financed college education for every student who passes the national exams. In brief, he argues that “Decent schooling, the absolute prerequisite to a decent life in America today, should be thought of as something that government guarantees to every citizen as a matter of right. It shouldn’t be left to local authorities to screw up, any more than flight safety should…. Test-prep should consist of mastering the high-school curriculum, not learning tricks to outwit multiple-choice aptitude exams.”

Did Lemann read all the pages leading to his Afterword? Those in charge of education policy on a national level are the beneficiaries of the current meritocracy, the same ones Lemann accuses of being more concerned for their own class than for the population at large. After the SAT admitted to testing teachable material instead of scholastic aptitude, test-prep did not disappear; on the contrary, it exploded. And his belief that government funded public higher education would alleviate the chasm in prestige between the upper echelon of (mostly) private colleges and the rest of the schools is laughable. Such an approach would return us to the beginning of Lemann’s narrative, where the Ivy League was more exclusive, more removed, and arguably more important, than it is today. His premises and his conclusions simply don’t match.

Even Lemann seems to realize that when he abandons those premises to find his backing in American history. “Universal opportunity has been a theme in American writing and fable and rhetoric at every point in our history. The current system was a much more particular, even tendentious development, which in ensuing years we have conflated with the general principle.” Setting aside the fact that the first sentence is not true, Lemann himself does plenty of conflating. While he never says the SAT is a bad test, he disdains its results, and thus dismisses the concept of a useful aptitude test. This is more a failure of his focus than of his thinking. The fact that the SAT is a crappy test does not mean that there can be no useful or good aptitude test. Indeed, Lemann’s conclusions are undermined by the fact that he never fully explores E.T.S.’s admission that the SAT is not really a successful measure of aptitude; he unfairly and inaccurately conflates all aptitude tests with the SAT — which isn’t an aptitude test.

Lemann makes a similar error in regards to equal opportunity. There is a difference between a society that values equality of opportunity and one that eschews social class completely. America has never been the latter. As Lemann would have it: “The founders of the American meritocracy . . . believe[d] they were destroying a nascent class system and
uilding a fluid, mobile society. In retrospect this was vainglorious — you can’t undermine social rank by setting up an elaborate process of ranking. Fifty years later, their creation looks more and more like what it was intended to replace.” The first part may be true; but only in as much as the meritocracy was conceived, created, and executed behind the scenes, without public knowledge, discussion, or input. Such is one of Lemann’s primary complaints — thus the “secret history’ of the subtitle — but such is the only way in which it could be said that the meritocracy was created to elide social class. Rather, the meritocracy aimed to replace a system that disbursed rewards arbitrarily with one that did so based on measurable merit in a competition open to all. This is very different from distributing those rewards completely equitably. Further complicating this qualification is another matter Lemann fails to discuss. The rewards distributed by the meritocracy are difficult to encapsulate: although Lemann doesn’t mention this, SAT scores and future income do not correlate. In fact, those who make the most money are almost always the white males with the lowest SAT scores at the most prestigious colleges. In other words, the meritocracy has not fully replaced the old boys’ network as the primary means of distributing rewards in our society. But Lemann blames the meritocracy for both its ostensible goals and its failure to reach those goals. In this book, the meritocracy can’t ever win.

Lemann’s final words are his truest. The meritocracy has, for many complicated reasons that are not all malicious, grown to resemble its predecessor in its exclusiveness, its seemingly hereditary nature, its apparent inevitability. But all too often, Lemann misses the big story in pursuit of the trivial detail, and elides reason in pursuit of support for his narrow agenda. The Big Test is an engaging story, but an incomplete one, and ancillary to the issues it truly seeks to address. It is as if Lemann wrote a long book as an excuse for his Afterword, which would have made a provocative (although flawed) magazine article.

As our initial analogy implies, we do not discard the Internet because of the limitations of our modems. So it is for the meritocracy. We should jettison the SAT; doing so would advance, rather than condemn, the cause of working toward an equal-opportunity, fluid, thriving, meritocratic society.

Contributing writer Jeff Mandell is really good at taking tests.