« Revisionist History

Lord of the Rankings

2021-07-01

For 30 years, US News & World Report has been using a secret formula to rank the best colleges and universities in the United States. As a public service to our listeners, we hack the algorithm and discover the dirty little secret of the rankings game.

Learn more about your ad-choices at https://www.iheartpodcastnetwork.com
This is an unofficial transcript meant for reference. Accuracy is not guaranteed.
Pushed him over 20th through November. Twenty eight Hamilton returns to Philadelphia tickets are on sale now for the Tony Alleviate Grammy and Pulitzer Prize winning musical coming to the Kimmel Cultural Campus. This fall. It's the story of America, then told by America now featuring a score that blends here. Pop jazz are and be an Broadway Hamilton has created a revolutionary moment and theatre performances, begin October, twentieth for six weeks only get tickets now a camel cultural campus dot org. No, is reading your mind but hp, instant, ink, When you're printers running low incense new cartridges before you run out, so you never have to think about in for details. Visit HP slash instant ink conditions apply, on a time in the middle of the nineteen eighties, when the cold war still ray
aged and a man named Ronald Reagan occupied the White House. There were three major weekly news: magazines in United States. Time magazine was number one with millions upon millions of copies in circulation. Newsweek was number two and what newsmagazine came third place, a sleepy little publication, based in Washington DC known as s news and world report one of the magazines managing editors in those years, Peter Bernstein, the day when everything changed, it was a very exciting time, because more had just taken over. There were a lot of people buzzing around more being Mortimer B, Zuckerman, a canadian billionaire who bought us news and what Report with dreams of becoming a media baron. The really good ideas in the magazine, and I think it's fair to say our feeling was the
the execution could have been a lot better and one the things we wanted to do was distinguished ourselves from time. Unusually, because we were the third How do you distinguish yourself if you all, was coming in. Third, if it can in billionaire with visions of ranger is breathing down your neck. You do something audacious. You create something that no educated parents could possibly ignore a, to examine every American College and University and rank them from top to bottom. We were looking to reinvent the magazine and the key drinkings became part of that so Bernstein and his colleagues began collecting no statistics on the universities and colleges of the United States to make sense of it numbers? They came up with an algorithm, a secret algorithm that could
sign every school, a score of one through one hundred, regardless of its size or mission or complexity. An algorithm to capture. The elusive quality of academic excellence. Finding those points of data and putting them into a ranking, as you can appreciate, is a complicated task. Yes, full of value judgments Full of problems, challenges dangers, but we were undaunted and proceeded I love you laughing about this. A very heavy time and we thought- and we did- we challenge time and news wicket their game. The rankings became an institution. U S. New surged careers were forged actions made me was fancied today is that time and music and basically disappeared
rest of, U S, news has basically disappeared, but the rankings strong as ever. Well, it's ice. The now in this world, of a firmer that something can last but and as a journalist, you know when your work is used for for old fish. After it a day it's nice, to see that something has an impact for a while Peter left us news long ago, but he and his colleagues had added something new to the American Academic Calendar beyond freshman week, finals, commencement there's another day. The new rankings come out. The new 30th edition of belt colleges was just released this morning and we've got it you're. Seeing the two thousand, the list for two thousand and nineteen was Princeton University, followed by Harvard Tide, had attention scholars America's best colleges
have been revealed the? U S, news and world report is out with its angle list and we ve got the inside scoop on the algorithm. This developed is undergoing a constant state of evolution over the years. How often do you tweaking it? I thought you might ask. I mean I did go online and look at it, and I was glad to see they are tweaking it on a regular basis, at least it looking. If you look at their methodology and of course they ve also been reacting to competition, I mean you know: a lot of people have been critical of the rankings and whether this mythology, really this methodology, exceed sends its leader. That is like a test, the definition of a Freudians you're talking about you, you're talking about you child, and you refer to it as mythology. That's true, I couldn't have come up with that. One consciously
at what point to the? U S: news methodology become a mythology. Let's find out My name is Malcolm global. Your listing to my podcast revisionist history, about things overlooked and misunderstood in this episode? I ask: how is it that a promotional gimmick from the 1980s turned into a fetish object for american higher education how do these rankings work? why does everyone take them so seriously broken. Launch a revisionist history investigation, one that will take us to all kinds of strange places to the bond markets, to pay And Oregon to New Orleans will be mysterious college administrator who ass to be identified only as D as well as a band of hackers who agreed to join our cause.
Alright, alright, so red goes very exciting. Now before we start can both of you guys introduced. So I'm and I'm an mcconville and I'm assistant, Professor Reed statistics at lying college, a number of international student who is currently studying at EU and sees grad school. So these are some of my accomplices,
and here's why I needed them? U S news will tell you what variables they used to determine whether a school is good or mediocre, but they won't disclose exactly how they combine those variables into a score. I mean by withdrawing its their secret sauce. They charge forty dollars a year for premium access to their findings. This is a big business, so I figured my only chance are really getting under the hood of the ranking. Algorithm was to find out if any one had in effect, hacked the algorithm and it turns out some one had. It was a group of students. It read a small liberal arts College in Portland Org under the leadership of their professor Kelly Mcconville, tell me how on earth this whole project of looking at US news rankings? How did it begin so? I teach a course called statistics practicum and in that course, then
its complete ass master, long empirical research project and I solicit those projects in the data from other red faculty and staff every year. U S news asks for a lot of data from schools, but the reader been estrangement had stop supplying the data they decided. They did. Believe in the rankings anymore, but still the magazine kept generating a ranking for red anyway and someone in the Red Administration came to Kelly Mcconville and said ever since we dropped out our. U S: news ranking has plummeted, we think they've been punishing us. Can you double check the math? That was our task. We were tasked with recreating the model and then seeing what our ranking would be if we did supplier data, yeah. So tell me the two: you walk me through. How does one go about conceptually trying to answer this question use a model, that's called awaiting some model. That means they take
twenty years, so different factors about a school like class size Graduation rates. They way each Those factors based on how important they think the factor is They sat upon those pieces and I'm calibrate it so that the highest course had a hundred and I'm based on those scores. They rank or schools the re team had to build their own version of the? U S news algorithm from scratch, tryin to get as close to it as possible, take out all the variables in kind from open governments ourselves and we built a model isn't trying to recreate the? U S, news, weight and some model. Instead, it's just trying to predict the: U S news
who are of a school a high degree of accuracy pointing ho ho close. Did you get Elsie further close, the almost school less outside of, like were reasonable range of predicted score in the test. Sad is read a really and all other schools are like with thing a reasonable prediction of all of their scores. During the red students, identical twins algorithm, is so good that when they run their rankings of american colleges, their list looks almost exactly like the. U S news list with one exception rate college weight, so they were messing with you Based on our finding, it seems reasonable they do something in twenty nineteen, you news put read in ninetieth place. Where do you think you should be ranked
Thirty, six: oh wait! What that much of a difference? Yeah! That's bananas, they, they really did a number on read. Now we don't to go much further into the ensuing beef between written newest news demand in calls the undergraduates analysis quote inaccurate, but just in my joy when I ran across weightings analysis here, I was searching for some way to understand the secret algorithm cooked up by Peterborough dignan his pals back in the nineteen eighties and lo and behold, a bunch of undergraduates it read college had cracked and where they motivated to help me absolutely because U S NEWS jist their school. My first question to Kelly and her team was: where should I start- and they said start with a little something called the
here assessment score? Can you give me a more precise sort of statistical sense of just how important is it like a car, so we find out pure assessment score is: has the largest they facts, so it has a coefficient of like six point. Fifty eight. In other words, if you hold everything else constant and you like increased, appear assessed. Scorer by one unit, your Roscoe will increase by six point. Five. Eight, the peer assessment score was at the hall part of the EU s new secret sauce, a measure of reputation, one of the most powerful elements in the whole ranking system so, if I'm a school- and I am anxious to move up the- U S news rankings the thing I should
we'd be focused on is trying to boost my reputation, score yeah yeah. I had my marching and I knew where I had to go next to the guardian of the: U S: news gates, the dragon at the mouth of the cave, Robert Horse, getting cash back, rewards for money, you're already spending who doesn't love. That would discover your cash back on repurchase, including your dinner for date, night new pajamas, that new streaming service or even festival, maybe that amazing new audio book from pushing industries spending more
she's this past year, one to spend a little extra on a loved one. The cash back you earn will be matched by discover automatically having a credit card that matches the rewards you already earned is one small thing you can do distress less about your finances and discover, makes it even easier because discover car is accepted at ninety nine percent of places that accept credit cards. So when it to discover get used to hearing yes, more often learn more discovered. A calm, slash, match limitations apply twenty twenty one. Nelson report would, if there was a sea
the hero who had the amazing power to make hiring faster, smarter and easier. Think of Zapruder like that money, the types of course, and we now you can try Zipper- could have her free only at the special place, zip recruiter, dot com, Slash Gladwell. Now what sort of hiring superpowers de zipper could have? Well, instead of fighting crime, I gotta superheroes, zip recruiter, fights time by helping you find qualified candidates fast. First, when you post your job, zip recruiters sends it out to over a hundred top jobs sites, with one click Tucker being in many places at once, then, zip recruiters matching technology, scours thousands of roses to identify people with the right skills and experience
for your job and actively invites them to apply. No wonder four out of five employers who post on the procurator, get a quality candidate within the first day, ready to conquer hiring, make sure you go to this special hiring superhero headquarters website zipper, cuter, dotcom, Flash Gladwell, let zip recruiter save the day, just go to zip recruiter, dot com, Flash Gladwell. Robert Morse, the man who runs the US news rankings, is an older man, white hair, very preppy, nice. If a bit lugubrious heavy hangs the head to wears the crown I called him up. I wanted to understand them Serious variable at the Red College hackers had told me was so important describe again. How is the reputation score.
Generated, so we send out three surveys school and then they respond. They rate the schools, on a scale of of five hundred and twenty one or no response to who are the three people at each school who are asked to put out the survey for the president the proposed and the roman manager or the head of emissions. The surveys arrive via email once a year we won who matters in american higher ed if you're out a liberal arts college you're, given a list of all two hundred and twenty two other liberal arts colleges in your category. If you're part of a big university, you get a list of the three hundred and eighty eight other big universities in the United States. Your job is to rank the schools on your list on a scale of one to five, five being amazing, one being the
the serious problem here and on what basis are people making those judgments roman amount to a question I mean we ask them to base it on our view of the schools, reputation for undergraduate academic quality, but it mean How would you know. I mean we believe that the aggregate, some of the all the raiders who were leaders in higher education in their presence, some progress, emission dean seller, not rookies when you're aggregate your views at any given time there representing where those schools stand in the marketplace mean it. So it's as it's an assessment of their feelings, what other schools, Well, they're leaders in higher education, so it's. On their knowledge of the other. Schools that that day
other or through meetings or for exposure they necessarily know a lot of all the schools are rating. I don't know If I read a restaurant on Yelp, it's because I have eaten there right gas, but here when RADA University on the US news, ranking it's not because I've attended that university or taught there is just well, you may have a region save that you didn't, but for saying have the people arrested the schools? every campers or target that school or no in depth their course catalogue, no Do some of them have greater degrees of knowledge about the schools are rating. Yes, the some of them. Have so degrees of knowledge? Yes, so I think it s very from having society
she'll knowledge to far less knowledge. In my native tat, I had imagined that there would be some level of rigour. It is crucial variable in the: U S: news, weight and some model, a methodology that I could study. Analyze, learn from, but Morse wasn't being very helpful. So I tried to draw him out. I mean I was first knew that there's it all clump, where why you Brigham young Yushiva and are all ranks right in the same roughly equally in the national, his rank, so a Mormon school conservative, Jouar School a Jesuit school and I was trying to think is the rabbi who runs yeshiva, know about Brigham Young and what does the Jesuit priest who runs can saga know about the Shiva No it's a good point like his gonzalvo giving.
China, five and giving Boston College of five and we're not looking at how people are rainwater schools, what we are doing, grim mean so we take off a few of the higher. Ratings and a few of the lowest ratings and in an effort to eliminate some of the strategic voting- that you're alluding to what are you doing, strategic voting on one or yours just it. If I am president of consortia, I'm powerfully motivated to believe that the Jesuit education is the best education out there. So when I look at Georgetown, I think there the right thing, and when I look at the sheep I say worse, the new testament her thin hut using the new testament, This is natural human biases, everybody to know what they. To do with the quality of the institution.
Well, I don't have a different answer voted listeners? Revisionist history will remember how years ago fell for Rowan University in New Jersey, a love ruin, but Rowan is ranked a hundred and eighty seven, in the National universities list dismal mediocre, and why? Because it's pure assessment score is a forgettable two point: four: how on earth have the university presence of America come to the collective decision that they hate, Rowan University Way way down and South Jersey? I call Allie hush Mud, the schools president, how many university presence had visited road in the last five years from other colleges two or three one from
One came who was positive factor when accreditation and I believe another war was afraid as it is so when there giving you a score of what basis today using for giving you How did they know anything at all of our own? Had other? I can only speak for myself. I can assure you that I really gone. No March evolve, they institution that I'm asked the right or school. I have. I just don't have enough those. Now, it's not like university president's, don't know anything about grading. They were all teachers themselves once, but when it comes to creating schools, it seems like they might just be mailing it in next. I call up the head of admissions at a major university. You ve heard of it trusts me, but I can't tell you what it is and we altered the voice to protect this person's identity. Of course we did. We had to look how? U S news treated Red College. My source told me-
to address them only as dean, I'm certain, I am far more qualified to be a member of the baseball writers, Association of Amerika and vote on the hall of fame than I. To assess other institutions for the: U S NEWS world report here I proposed to deem that we do and exercise I'd name a school at random and then the two of us. We try to peer, assess it floored estate, never dinner, given I've got in there. I think people to Syria Yes, I I for coffee there once in a way driving back to my mom's house in Canada. I don't know that counts, don't think it does
at the house, you gonna next periods, I gotta sex, various the weather, but there's? No, there were no students, there was Christmas, but I will say this: my cousin went to Syracuse and had a good time. So an answer you read it based on that: let's give Syracusan love three point: nine, no foreplay, now diversity of washing in Seattle, yes, winner, yes, I have, would you do when you there I walked around. I think the council accounts I said dean. You can see the mountains from there the space needle let's go big University of Washington at Seattle we're point to have you ever been William Mary? I have I love.
Of history and love Colonial Williamsburg. Do you remember what school you gave him a merry? I am. We gave them a four or five. Really you went for five, second notice college in the? U s you think that affected your judgment on it, I'm sure, did didn't. I went on for quite some time we'll along shout about Southern Methodists University in Dallas and whether or not the great football team School had in the nineteen eighties. Should counterfeit Dean said no. I said yes now was all this awful exercise not really, but then again not. It was my conversation with the visit of the rankings, Robert Morse, one suggestion. That has been made to me by people who do the rankings. Is it the thing they rely on to make their reputation score for other universities. The only source they have of reputation, scores
The other universities, which is to previous years, use news rankings. Is this circular If they're telling you that an that's our expectation of how people are The ratings risk is looking at the ranking table in and looking at the score. We believe that there is more thought, Going into it than that, but we haven't any kind of social science, research to prove or or disprove that point you haven't any right: yeah they haven't done any social science research. Only one person gave me much in I entered the mechanics of the Peer assessments core and that was alley hush mud president of ruin. His advice was, you have to take matters into your own hands, so what does he do? He makes his own house
sauce and he told me that Rowan sends out free samples to every college president in a country as such, a rethink their hands at present, now know that I make hot sauce. That would help me cooling, I hope Wait. I want to know about your hearts us what kind of hearts ass to her. Oh, I make serious serious hearts us, but I actually grow from seeds all the way to the packaging, and there are three levels of eat: Lois, One is called alleys nasty, not just kind of my in the one next to it, with happens as a seat is called as delicious and then the on with colored progress, nasty vicious and I will send you a package, the nasty, Can I get you Is it going to help your cause? Is gonna? Hurt you guys if you're you're, giving some
university present in the middle of the West is like TAT happens to have a low, but a vicious circle, digging a curse. Name. But what I figured that no no publicity about about policy. Did I tried a nasty vicious. Oh, yes, I did on a scale of one to five five point.
What if there was a superhero who had the amazing power to make hiring faster, smarter and easier. Think of Zapruder like that money, the types of course- and I now you can try Zipper- could have her free only at the special place, zip recruiter, dot com, Slash Gladwell. Now what sort of hiring superpowers de zipper could have? Well, instead of fighting crime, I gotta superheroes, zip recruiter, fights time by helping you find qualified candidates fast. First, when you post your job, zip recruiters sends it out to over a hundred top job sites, with one click Tucker being in many places at once, then, zip recruiters matching technology, scours thousands of roses to identify people with the right skills and experience.
For your job and actively invites them to apply. No wonder four out of five employers who, post on the procurator, get a quality candidate within the first day, ready to conquer, hiring, make sure you go to this special hiring superhero headquarters website zipper, cuter, dotcom, Flash Gladwell, let zip recruiter save the day, just go to zip recruiter D. Come Flash Gladwell October twenty through November twenty eight Hamilton returns to Philadelphia tickets or on sale. Now for the Tony alleviate Grammy an Pulitzer Prize winning musical coming to the Kimmel Cultural Campus, this fall. It's the store
of America, then told by America now featuring a score that blends hip hop jazz, r and D. An Broadway Hamilton has created a revolutionary moment and theatre performances, begin October, twentieth for six weeks only get tickets. Now a Kimmel Cultural Campus, dot org. In the middle of my investigation of the pure assessment score and its many implications. I had a conversation with a professor at Duke named bill. Mayo may was friendly with some bond Peters and he was talking with some of them one day when they mention that they were having difficulty with bonds issued by historically black colleges, H, BC use as their no if you're a bond trader. You worry about me the bonds that you have an inventory and they just me
and you know- Hey- ruin a really hard time placing these HBO. You schools, because people don't want to hold on the way the bond market works. Is that an institution needs money. Let's hear university wants to build a dormitory, the cost, fifty million dollars the school can't raise all that money on its own. So it goes to a Wall Street Bank which purchases a bond, a promise of repayment from the university. Then the bank breaks that bonded too many hundreds and hundreds of pieces and sells those pieces to wealthy investors or insurance companies or headphones. So in may you hear that banks are struggling to sell bonds for historically black colleges. He wonders is just some apocryphal Wall Street story, or is it true? Luckily, is a simple way to test this. The price the bond trader charges is called
gross spread. If it's super easy to find buyers for the bonds, the gross spread is likely to be low. If the trader has difficulty finding buyers, the spread can get quite pricey. So if the bonds from black schools are harder to move than bonds from white schools, the spread should be higher and are they? Yes, they are Mau and his colleagues compared black schools with Non Hbcus that were otherwise identical. The spreads for the historically black schools were twenty percent higher in the bond market? A twenty percent difference in spreads is huge mayo and his co. Researchers then went one step further They zero in on three states in the deep south and you those these states were
so those are Louisiana, Mississippi and Alabama, and what's the difference in spread in those states, not twenty percent? Thirty, seven percent. May you and his colleagues have given us a scientifically rigorous measurement of just how much of a burden black instead Sean's face in the real world. You ve put a number on reputation. What is the reputational cost of being a school that overwhelmingly accepts, accepts and educates blockades, in Louisiana. The reputation cost is thirty percent. That Way to think about it, but I would say it's the racial component of reputation. A spread is a reputation score. Isn't it invests? just don't know the precise ins and outs of all the institutions are asked to buy. How could they? There are literally hundreds of thousands of bonds and a market at any given time, so the investors proceed on a feeling
a feeling about how worthy and how safe and held legitimate the institution? during the loan. Is you see a black school and you say to yourself: I dont value that each pc you quite as highly is the whites could across town. So you give the black school a discount, which is what prejudice is by the way, a discount the black life. Does not matter as much as the white life and who are these people
while imposing a discount on black schools there, wealthy successful people who are simply checking a box. I like this bond holocaust when a lot I dont like this one. They are, I suspect, not a whole lot different from the college administrators who sign a number today appears for the: U S: news rankings every year, motivated by the same unconscious biases I liked the school forty two I like this one alot four point: five. I dont like this one two point four, even though I don't really know anything about it. So if you are the president of a historically black college in one of those states with reputational discount for being black, is thirty seven percent. How do you think it feel about you appear assessment score
I thought I'd call water Kimber, president of Dillard University, a historically black college in Louisiana, and so when I start to look at the formulas and dig into it, I was like this isn't a real good measurement of what we do It is a measurement, a privilege, Dillard reputation. Score is two point. Six has not good. You can ever get to the top of the. U S: news rankings with appear assessments core of two point: six. Maybe that sounds defeated. But it's not your readers, its realistic! Israel is a living it now. He's. Gonna just Goin say this with all these black people is great in their haste, to compete with time and Newsweek way back maybe all U S. News did was created and that allowed the president's Indians of every college in a country to assign a number to their prejudice to disguise mythology as
Theology next week on revisionist history, We take up the cause of Devils University and I lost my own friends at RE college to help me out So that is how we can dealer. Into the tippy top tier of the liberal arts colleges. So Williams, Amherst dealer Swarthmore gas news history is produced by me: Label Leman, Gusto and Jacob Smith, with Elouise Linton and on an eye, our editor, is Julia Barton, original scoring by Luis Cara, mastering by Flan Williams and engineering by Martin Gonzalez fact, checking by Amy gains. Special thanks. Pushkin crew had a thing hardly make liore my a caning Taylor, Eric Sandler, Nicole Moron
Jason, Gambrels and, of course, Jacob Iceberg enough of that? If you love this show and others from Pushkin industries consider becoming a push. Nick push Nick is a podcast subscription that offers bonus, content and uninterrupted listening for four dollars and ninety nine cents a month look for push Nick exclusively on Apple, podcast subscriptions it three flavors allies nasty delicious and nasty vicious ok, nice, we alone wedges with do something, as they get one two three
yeah it's coming harder. It comes on, it's glad and steady. Oh, I do that's it it's go away. This is really good I, for a very long time, because, Snuff is hard. the second like this, but in struggling What you try my Harvard? How sounding sound out see any Harvard? Has us one sent to me hot sauce from the northern. Now over 20th through November twenty eight Hamilton returns to Philadelphia tickets are on sale. Now for the Tony Alleviate Grammy and Pulitzer Prize winning musical coming to the Kimball Cultural Campus, this fall. It's the story
of America then told by America now featuring a score that blends here pop jazz r and D, an Broadway Hamilton has created a revolutionary moment and theatre performances, begin October, twentieth for six weeks only get tickets now a camel cultural campus dot org. Spending more and groceries this pasture wanting to order some weekday take out
whether you're spending on the essentials, like groceries or splurge, on dinner from your favorite local restaurant, do so with your discover credit card because only discover automatically matches all the cash back. You you've earned at the end of your first year and it is so easy to earn cash cashback since discover card is accepted at ninety nine percent of places in the US that take credit cards when it comes to discover, get used to hearing yes, more often learn more at discover. I come slash, match imitations apply. Twenty twenty one. Nelson report,
Transcript generated on 2021-08-14.