The Amazing Truth About PISA Scores: USA Beats Western Europe, Ties with Asia

Once we correct (even crudely) for demography in the 2009 PISA scores, (PISA is the Program for International Student Assessment) American students outperform Western Europe by significant margins and tie with Asian students. Jump to the graphs if you don't want to read my boring set-up and methodology.

The main theme in my blog is that we shouldn’t confuse policy with culture, and with demographic factors.

For instance, education scholars have known for decades that the home environment of the kids and the education levels of the parents are very important for student outcomes. We also know that immigrant kids have a more difficult time at school, in part because they don’t know the language.

Take me as an example. The school me and my brother attended was in a basement in Tehran, had no modern resources, and largely focused on religious indoctrination. But we had a good home background. Our father attended a college in the west a few years (our mother didn’t, despite stratospheric scores test scores, because at the time you didn’t send a good Kurdish girl to another city to study). So we did well in school. Conversely, the first few years in Sweden I had bad grades, in part because I didn’t master the language.

The point I am trying to make is that the school in Sweden was objectively superior to the school in Iran. But I scored lower in Sweden, because of factors outside the control of the education system. If you want to compare the effect of the school, you have to isolate those external factors and make an apples-to-apples comparison.

However, this is not at all how the media is presenting the recent PISA scores. For example there is a lot of attention of the score of the kids in Shanghai, the according to the NYT is supposed to “stun” us or something.

It's dumb to compare one of the most elite cities in a country with entire nations, and to draw policy-inference from such a comparison. Shanghai has 3 times the average income of China! It is also naive to trust the Chinese government when they tell us the data is representative of the entire nation. Either you compare Shanghai to New York City, or you compare the entire country of China, including the rural part, with other large nations. Most of the news and policy conclusions we read about PISA-scores in the New York Times is thus pure nonsense.

1. Correcting for the demography:


In almost all European countries, immigrants from third world countries score lower than native born kids.

Why? No knows exactly why. Language, culture, home environment, income of parents, the education level of the parents and social problems in the neighborhood and peer groups norms are among likely explanations. But it is generally not true that the schools themselves are worse for immigrants than natives. In welfare states, immigrants often (thought not always) go to the same or similar schools and have as much or likely more resources per student.

So the fact that immigrant students in mixed schools do worse than Swedish kids used to a few decades ago in homogeneous schools does not it out of itself prove that Swedish public schools have become worse.

Of course, the biggest myth that the media reporting of PISA scores propagates is that the American public school system is horrible.

The liberal left in U.S and in Europe loves this myth, because they get to demand more government spending, and at the same time get to gloat about how much smarter Europeans are than Americans. The right also kind of likes the myth, because they get to blame social problems on the government, and scare the public about Chinese competitiveness.

We all know that Asian students beat Americans students, which "proves" that they must have a better education system. This inference is considered common sense among public intellectuals. Well, expect for the fact that Asian kids in the American school system actually score slightly better than Asian kids in North-East-Asia!

So maybe it’s not that there is something magical about Asian schools, and has more to do with the extraordinary focus on education in Asian culture, with their self-discipline and with their favorable home environment.

There are 3 parts to the PISA test, Reading, Math, and Science. I will just make it simple and use the average score of the 3 tests. This is not strictly correct, but in practice it doesn’t influence the results, while making it much easier for the reader. (the reason it doesn't influence the results is that countries that are good at one part tend to be good at other parts of the test.)

The simplest thing to do in order to get an apples-to-apples comparison is to at least correct for demography and cultural background. For instance, Finland scores the best of any European country. However first and second generation immigrant students in Finland do not outperform native Swedish, and score 50 points below native Finns (more on this later).

On PISA, 50 points is a lot. To give you a comparison, 50 points is larger than the difference between Sweden and Turkey. A crude rule of thumb here is that 50 points is 0.5 standard deviations.

The problem is that different countries have different share of immigrants. Sweden in 2009 PISA data had 17%, and Finland 4%. It’s just not fair to the Swedish public school system to demand that they must produce the same outcome, when Sweden has many more disadvantaged students. Similarly schools with African-American students who are plagued by racism, discrimination, crime, broken homes, poverty and other social problems are not necessarily worse just because their students don’t achieve the same results as affluent suburbs of Chicago. In fact, the most reliable data I have seen suggests that American minority schools on average have slightly more money than white schools. It’s just that the social problems they face are too much to overcome for the schools. It is illogical to blame the public school system for things out of its hands.

So let’s start by removing those with foreign background immigrants from the sample when comparing European countries with each other. I define immigrants here as those with a parent born outside the country, so it includes second generation immigrants. This is fairly easy for Europe.

In the case of America, 99% of the population originates from other countries, be they England, Italy, Sweden, India, Africa, Hong-Kong or Mexico. If we want to isolate the effect of the United States public school system, we should compare the immigrant groups with their home country. For those majority of Americans whose ancestors originate from Europe, we obviously want to compare them with Europe. For some groups, such as Indians, this is inappropriate. The reason is that mainly the most gifted Indians get to migrate to America to work or study.

However, as I have argued previously, there is strong reason to believe that this problem of so called biased selection does not apply to historic European migration to the United States at the aggregate level. The people who left Europe were not better educated than those who stayed. Immigrants were perhaps more motivated, but often poorer than average.

So similar to my comparison of GDP levels, let us compare Americans with European ancestry (about 65% of the U.S population, and not some sort of elite) with Europeans in Europe. We remove Asians, Mexicans, African-Americans and other countries that are best compared to their home nations. In Europe, we remove immigrants.

The results are astonishing at least to me. Rather than being at the bottom of the class, United States students are 7th best out of 28, and far better than the average of Western European nations where they largely originate from.




The mean score of Americans with European ancestry is 524, compared to 506 in Europe, when first and second generation immigrants are excluded. So much for the bigoted notions that Americans are dumb and Europeans are smart. This is also opposed to everything I have been taught about the American public school system.

For Asian-American students (remember this includes Vietnam, Thailand and other less developed countries outside Northeast Asia), the mean PISA score is 534, same as 533 for the average of Japan, South Korea, Singapore and Hong Kong. Here we have two biases going in opposite directions: Asians in the U.S are selected. On the other hand we are comparing the richest and best scoring Asian countries with all Americans with origin in South and East Asia.


2. Policy-Implications


Libertarians in the United States have often claimed that the public school system (which has more than 90% of the students) is a disaster. They blame this on government control and on teachers unions. However, it is completely unfair to demand that a public school in southern California where most of the students are recent immigrants from Mexico whose parents have no experience in higher education (only 4% of all Mexican immigrates have a college degree, compared to over 50% of Indian immigrants) should perform as well as a private school in Silicon Valley.

The libertarians have no answer why European and Asian countries that also have public school systems score higher than the United States (unadjusted for demography). Top scoring Finland has strong teacher unions, just as California.

Similarly, the left claims that the American education system is horrible, because Americans don’t invest enough in education. The left has no answer when you point out that the United States spends insanely more than Europe and East Asia on education. According to the OECD, the United States spends about 50% more per pupil than the average for Western Europe, and 40% more than Japan.



Another policy implication is that Europe can learn from American public schools, which appear to be better than most European countries. I can only compare Sweden with the U.S, but I can tell you that from my experience, the American system is superior. I always thought this was just anecdotal evidence, but I am beginning to realize that American schools are indeed better.

For example, we don’t have any real equivalent to Advanced Placements classes. We have cheaper and worse textbooks. The teachers on average have far less education. I could go on.

Nor is it any longer a mystery to me why Americans spend so much more on education and (falsely appear) to get out less in output.

But of course the biggest implication is that most Europeans and all American liberals have lost the bragging right about their side being smarter than Americans.

3. Immigrant PISA scores compared to natives


This is again the mean difference of the 3 parts of PISA.


Australia is the only country with a negative gap, which means Australian immigrants actually score better than natives. Canada is similar. The Australian-Canadian skill based migration system is at work here, generating less inequality (even short term).

The other pattern appears to be that the gap is almost constant in the remaining Western European countries. This may be important to keep in mind, whenever people claim that uniquely Swedish policies are causing poor immigrant educational outcomes.

Tino Sanandaji is of Kurdish origins, and was born in Iran in 1980. In 1989 he moved together with his mother and brother to Sweden. He has a degree from the Stockholm School of Economics, his M.A in Economics from the University of Chicago and is expected to receive his PhD in Public Policy from the University of Chicago in 2011. His work has appeared in The Wall Street Journal, The New York Times, National Review, and numerous Swedish newspapers. Tino has been a resident of Hyde Park Chicago, since 2004.

Subjects:

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

This is a great inspiring

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information...
Tacoma String Quartet Seattle

Fantastic blog you have

Fantastic blog you have here. You’ll discover me looking at your stuff often. Saved!
energy watch

Knowledge and immigrants

There are no bad schools, there are lazy kids that don`t want to learn at least 60% of their capabilities. My Health Information Technology degree is not so relevant in PISA scores, I read all this post with great interest, Mr.Tino Sanandaji wrote some interesting facts that I did not think they were so crucial. Once again we discover that confusing "policy with culture and demographic factors" can lead to some results that don`t reflect the "reality", for good results all these aspects must be put into consideration. After all, it`s the new geography, right?

Hi

This is great, I love to see kids who really study and want to do something great with their lives.I was reading something about a new job networking and I bet that the job want end up in the hands of somebody who didn't study hard during his schooling period.

After reading your article,

After reading your article, I am quite baffled to be honest. I have changed my view on how choosing a career is easy. It is surely not. It can prove to be quite hard to get to the step where you can actually make that choice.

Interesting statement

"The liberal left in U.S and in Europe loves this myth, because they get to demand more government spending, and at the same time get to gloat about how much smarter Europeans are than Americans. The right also kind of likes the myth, because they get to blame social problems on the government, and scare the public about Chinese competitiveness."

Having been a Secondary Chemistry teacher (as well as a Geographer) I am fascinated by this assessment. The idea that this is used to justify "more" goverment spending. I am not sure what is meant by "more". I worked in a district that has not passed a working budget in the last several years - with the argument always the same: the lazy, (or whatever adjective applied) teachers do not deserve more. But when you get right down to it - teaching of science especially costs money. Lab materials and chemicals are not cheap and State mandates do not usually fund themselves. And the right "kinda" liking this myth - I say they wrap themselves in it and use it to blame teachers (and only teachers) for issues related to school performance.

I agree with the normalization of this data with demography. However, I will caution you to be careful in how you characterize who uses what information to cast blame. As a former teacher I am often *outraged* by the characterization of the issue as an either/or; with teachers as the *always* the main culprit. Until you have walked a mile...

I taught in some really,really tough situations (both urban poor and rural poor) and invariably when I wanted to do more with my teaching I often reached for my own wallet to fund it. That is the real shame of this - no trust anymore for teachers and public schools.

Every system has waste in it - but the austerity with which we treat US education is shameful.

poor methodology

The following article applies better methodology. While it does not attempt to factor out immigrants, it does compare other demographic performance (e.g. white students and students with a college educated parent). http://educationnext.org/teaching-math-to-the-talented/

I have read at least 50

I have read at least 50 different commentaries in the popular media, often by so-called "experts" which make reference to these scores. Tino Sanandaji is the most intelligent commentator I have encountered. Mr. Sanandaji quite obviously understands statistics better than people who are writing columns for The Washington Post and New York Times!!! Mr. Sanandaji understands complex interactions and relationships between current immigration flows, history, economic status, and culture. He has a wide life experience, obviously, that enables him to approach issues of immigration with sensitivity and insight. I do not know what he is doing for a living, but I certainly hope that President Barack Obama will find this person and offer him a job immediately. Immigration in the United States is such a serious conflict that we need someone who can help us find a peaceful solution that is fair to all parties involved. I know that Mr. Sanadaji is writing about the education issue only. When you're really smart, Mr. Sanadaji, it's hard to hide it. Thank you for your efforts to be fair and just. If more people tried to do that, rather than just run their lives by greed and self-interest, the world would be a better place.