Ítems contextualizados de PISA Ciencias: La conexión entre las demandas cognitivas y las características de contexto de los ítems

Maria Araceli Ruiz-Primo; Min Li

doi:10.7203/relieve.22.1.8280

Authors

Maria Araceli Ruiz-Primo Stanford University
Min Li University of Washington

DOI:

https://doi.org/10.7203/relieve.22.1.8280

Keywords:

PISA, science items, context characteristics of the items, cognitive demands, validity.

Abstract

The ubiquitous use of contexts in test items is based on the premise that contextualizing items is an effective strategy to test whether students can apply or transfer their knowledge. In this paper, we continue a research agenda focusing on testing this premise. We present a study of the context characteristics in a sample of 2006 and 2009 PISA science items and how these characteristics as well as student performance may be related to the cognitive demands of the items. The study was guided by two research questions: (1) What are the cognitive demands of the sampled PISA contextualized items and what is the students’ performance linked to these items? (2) Are the items’ cognitive demands associated with certain characteristics of the contexts of the items that proved to be linked to students’ performance? Using 52 released and secured PISA science items, we captured information about three context dimensions of items, level of abstraction, resources, and nature of the context, and the cognitive demands of the items. A multinomial logistic regression with cognitive demand as the outcome variable, context characteristics as the predictors, and percent of correct responses as the covariant indicated that certain context characteristics are linked to the cognitive demands of items. For example, we found that items in which contexts involve only concrete ideas were associated with items with low cognitive demands; these items are unlikely to require content knowledge to be responded. We also found that the type of resource (e.g., tables, graphs) was associated with the cognitive demands of the items: schematic representations seem to be linked to items tapping procedural knowledge rather than to items tapping declarative or schematic knowledge. We concluded that further research is needed to better understand the influence that context characteristics have on the cognitive processes in which students are asked to engage and in their performance.

Key-words: PISA, science items, context characteristics of the Items, cognitive demands, validity.

Downloads

Download data is not yet available.

Author Biographies

Maria Araceli Ruiz-Primo, Stanford University

is an associate professor at the Graduate School of Education at Stanford University. Her work, funded mainly by the National Science Foundation and the Institute of Education Sciences, examines the assessment of student learning both in the classroom and in large-scale assessment programs and classroom assessment practices. Her publications address the development and evaluation of diverse learning assessment strategies (e.g., concept maps and students’ science notebooks), and the study of teachers’ informal and formal formative assessment practices (e.g., the use of assessment conversations and embedded assessments). Her recent work focuses on the development and validation of assessments that are instructionally sensitive and instruments intended to measure teachers’ formative assessment practices. Her address is: Graduate School of Education. 485 Lasuen Mall. Stanford University, Stanford, CA

Min Li, University of Washington

is associate professor at College of Education, University of Washington, Seattle. She is interested in understanding how student learning can be accurately and adequately assessed both in large-scale testing and classroom settings. Her work reflects a combination of cognitive science and psychometric approaches in various projects, including examining the cognitive demands of large-scale science items, using science notebooks as assessment tools, parameterizing the design of contextualized tasks, analyzing teachers’ classroom assessment practices, and validating complex performance-based tasks for teachers and educational leaders. Her address is: University of Washington ? College of Education. Seattle, WA 98195-3600.