How not to reason with PISA data: an ironic investigation

International comparative testing, such as the OECD Programme for International Student Assessment (PISA), has considerable impact on policy-makers, the media and the general public. A central assumption underlying PISA is that global variation in students' academic performance is attributable to national educational structures and policies. The aim of this article is to demonstrate the problematic nature of this assumption. Rather than critiquing it from the outside, we turn the tools, data and presumptions of the current discourse of international comparisons upon themselves, showing that this assumption is refuted by analysis of immigrant student test scores. Data from the PISA 2009 tests show that Chinese immigrant students in New Zealand and Australia achieve math scores that are more similar to those of students in Shanghai than to their non-immigrant Australian and New Zealand peers. Thus, cultural background appears to be more consequential for the educational attainment of Chinese immigrant students than exposure to the educational systems of Australia or New Zealand. We discuss limitations of our analysis, using them as basis to discuss the shortcomings of PISA more generally.