Correcting for Scale Usage Differences among Latin American Countries, Portugal, and Spain in PISA

Abstract

This paper investigated the effects of corrections for scale usage preference in seven Latin American countries, Portugal and Spain in student self-reports in the 2012 Programme for International Student Assessment (PISA). These targeted countries tend to show a strong tendency of expressing strong opinions and self-enhancement, which can pose serious validity threats in cross-cultural comparisons of self-reports. We examined to what extent score corrections, that have been proposed, would change the patterning of the cross-cultural differences. We corrected for the scale usage preferences in a measure of teacher support among 39,045 students in nine countries, based on extreme response style, overclaiming, and anchoring vignettes, respectively. These measures showed different effects:  (1) All correction methods helped to improve measurement invariance, although the correction based on anchoring was less effective in reaching scalar invariance compared with the correction of extreme response style and overclaiming; (2) controlling for extreme response style and overclaiming changed the mean score of Spain to a greater extent than other countries, which seems to present a more realistic patterning, whereas the changes on correlations with other measures were rather limited. The use of anchored scores led to drastic changes both in means and correlations. A firm conclusion about which method is to be preferred cannot be given as there is no evidence which method enhances the validity of scores in these countries more. We discuss the necessity and practicability of correction methods.