Estudio cuantitativo multidimensional de la experiencia de evaluación 360 encuestada en prácticas de ingeniería industrial

Francisco Javier Granados Ortiz; Ana Isabel Gómez Merino; Jesús Javier Jiménez Galea; Isidro María Santos Raez; Juan Jesús Fernández Lozano; Jesus Manuel Gómez de Gabriel; Joaquín Ortega Casanova

doi:10.30827/relieve.v29i1.25356

Authors

Francisco Javier Granados Ortiz University of Almería https://orcid.org/0000-0002-7160-7294
Ana Isabel Gómez Merino University of Málaga https://orcid.org/0000-0002-0840-3985
Jesús Javier Jiménez Galea University of Málaga https://orcid.org/0000-0002-2223-695X
Isidro María Santos Raez University of Málaga https://orcid.org/0000-0002-6502-8499
Juan Jesús Fernández Lozano University of Málaga https://orcid.org/0000-0002-8174-1331
Jesus Manuel Gómez de Gabriel University of Málaga https://orcid.org/0000-0001-5960-3453
Joaquín Ortega Casanova University of Málaga https://orcid.org/0000-0001-7660-6875

DOI:

https://doi.org/10.30827/relieve.v29i1.25356

Keywords:

Survey, satisfaction, confirmatory factor analysis, industrial engineering

Abstract

Initiative and proactiveness shown by students during engineering lectures is usually very limited. However, students usually show high levels of interest in practical laboratory sessions. In order to address increasing dropout from engineering courses, as well as decreased enrollment, the present study aims to quantitatively analyze the impact of a 360-degree feedback survey for evaluating practical sessions. Analysis was conducted overall and as a function of industrial engineering students. Several objectives are intended to be achieved. Firstly, the study aimed to engage students in the evaluation process and, secondly, identify satisfaction with 360-degree feedback as a function of different groupings, whilst, at the same time, gathering opinions about the fairness of evaluation. To this end, a methodology based on the application of 360-degree feedback was applied and a 23-question survey was administered. The following three stages were followed for the 360-degree feedback evaluation process: co- (between students), self- (the student themself) and hetero-evaluation (lecturer). Initially, a questionnaire was designed and validated using confirmatory factor analysis. Responses were analyzed as a function of 4 groups: module (one first- and one third-year module), evaluation type, sex (male or female) and degree level (BSc or MSc). The most appropriate weighting to be applied to each evaluation in order to produce a final overall score was also analysed. This suggested optimal values of 50%, 30% and 20% for the hetero-, co- and self-evaluations, respectively. Additionally, outcomes revealed a high degree of satisfaction for all analysed groupings and high level of maturity in participating students.

Downloads

Download data is not yet available.

Author Biographies

Francisco Javier Granados Ortiz, University of Almería

Ramón y Cajal research fellow in the Department of Engineering, University of Almería (Spain). Industrial engineer awarded by the University of Málaga and PhD in Computing and Mathematical Sciences awarded by the University of Greenwich (UK) in 2017. He was Marie Curie Fellow within the European project FP7 AeroTraNet2 project, where he worked together with 9 prestigious institutions across Europe from both industry and academia. After this project, he worked as Data Scientist in industry for three and a half years. In 2020 he joined to University of Malaga as research engineer and PAIDI (Andalusian Plan for Research, Development and Innovation) postdoctoral fellow and he currently holds a Ramón y Cajal excellence research fellow at the University of Almería (Spain). He has published several research papers in renowned journals (20+ publications as first author), carried out research stays in 8 different research institutions, developed academic activities (teaching in Engineering schools, invited lecturer in University of Málaga), and attended many conferences worldwide (25+ presentations). He has participated in two education projects and he is accredited as lecturer and senior lecturer by ANECA.

Ana Isabel Gómez Merino, University of Málaga

Professor (CEU) in the Department of Applied Physics II, University of Malaga (Spain). PhD in Physical Sciences from the University of Granada. Member of the Royal Spanish Physics Society. She has worked on Educational Innovation Projects, delving into aspects related to the use of ICT Technologies in Physics learning, the influence of tutorials on the teaching-learning process, women in Engineering, interdisciplinary coordination and interactive evaluation. She has made stays at the Universities of Skövde (Sweden) and Albuquerque (USA), the latter with the ISTEC group (Ibero-American Science and Technology Education Consortium).

Jesús Javier Jiménez Galea, University of Málaga

Interim Substitute Professor in the Department of Civil Engineering, Materials and Fabrication, University of Malaga (Spain). PhD at University of Malaga in Mechatronic Engineering. Industrial Technical Engineer specializing in Mechanics, Technical Engineer in Industrial Design, University Master's Degree in Occupational Risk Prevention, University Master's Degree in Educational Innovation and Knowledge Management from the University of Malaga. Membership of the Quality Assurance Commission of the Doctoral Program in Mechatronic Engineering. Researcher at PIE17-165, at PIE19-032 and at PIE22-094. Membership of the PAIDI FQM231 group. Coordinator of various courses at the General Foundation of the University of Malaga (FGUMA). Collaborator in 2016 in "Las 101 mañanas" of the 101tv network. From 2016 to the present, he is a teacher and researcher as an Interim Substitute Professor at the School of Industrial Engineering of the University of Malaga.

Isidro María Santos Raez, University of Málaga

Assistant Professor in the Department of Mechanical, Thermal and Fluid Engineering, University of Malaga (Spain). Professor of Secondary Education in the specialty of Organization and Processes in Vehicle Maintenance, Andalusia Government. Assistant Professor, in the Area of Thermal Machines and Engines, at the School of Industrial Engineering, University of Malaga. PhD Industrial Engineer, Technical Industrial Engineer from the Polytechnic University School and Engineer in Industrial Organization from the Higher Technical School of Industrial Engineering, University of Malaga. Certificate of Pedagogical Aptitude and Higher Technician in Occupational Risk Prevention. Researcher in Teaching Innovation Projects related to multidisciplinary, horizontal and vertical coordination, peer evaluation and use of ICT technologies. Co-author of scientific articles in the field of heat engines, nanofluids and teaching methodologies.

Juan Jesús Fernández Lozano, University of Málaga

Associate Professor in the Department of Systems Engineering and Automation, University of Málaga (Spain). J.J. Fernández-Lozano obtained the Master and PhD degrees in Industrial Engineering from the University of Málaga, in 1997 and 2002, respectively. From 2004 to 2012, he was Vice-Director of the School of Industrial Engineering of the University of Malaga, and Director, from 2012 to 2016. He has participated in the design and implementation of new bachelor's and master's degrees in accordance with the European Higher Education Area. In 1998 he joined the Robotics and Mechatronics Group. Since 2009 he is Associate Professor in the Department of Systems Engineering and Automation. Besides innovation in higher education, his main research interests include applications of robotics, mechatronics and intelligent control to different fields, from intelligent vehicles to industrial applications or emergency missions. In addition to his activity as PI and Researcher for publicly funded research projects, he collaborates regularly with different private companies. As a result of these activities, in addition to authoring articles in journals and conferences, he has filed 11 patents, seven of which have been transferred to industry.

Jesus Manuel Gómez de Gabriel, University of Málaga

Associate Professor in the Department of Systems Engineering and Automation, University of Málaga (Spain). Ph.D. In Computer Engineering from the University of Malaga (1999) Associate Professor in the Department of Systems Engineering and Automation. He has researched on mobile robotics, surgical robotics, and engineering education. He is currently developing his research on the physical interaction between robots and humans. He has made stays and collaborates as a reviewer and guest editor of international journals. Co-director of four doctoral theses on various forms of interaction between robots and humans. Principal Investigator of two research projects with National Funds and others of a regional and local nature. Has been a leader of four Innovation Research Projects and participated in international summer courses.

Joaquín Ortega Casanova, University of Málaga

Full Professor in the Department of Mechanical, Thermal and Fluid Engineering, University of Malaga (Spain). Industrial Engineer awarded by the University of Málaga, becoming part of the first group of students of the former School of Industrial Engineering (1990-1995). After that, he was an FPI (Research Staff Training program grants) fellow associated with a Spanish Government Project (1996-1999) and defended the PhD Thesis in the field of Fluid Mechanics at the University of Malaga in 2000. He was Assistant Professor from 1999 to 2003, Associate Professor from 2003 to 2021 and Full Professor from March, 2021. He is currently Academic Coordinator of the Degree in Industrial Technology Engineering from 2012 at the University of Malaga. He has completed two pre-doctoral stays at the Arizona State University (Az, USA), and different post-doc stays at the University of Greenwich (London, UK) and the University of Porto (Porto, Portugal). He was Deputy Vice-Chancellor for Technology Transfer from 2018 to 2020, and since then, he is Deputy Vice-Chancellor of Infrastructures at the University of Malaga.

References

Abadía Valle, A. R., Bueno García, C., Ubieto-Artur, M. I., Márquez Cebrián, M. D., Sabaté Díaz, S., & Jorba Noguera, H. (2015). Competencias del buen docente universitario. Opinión de los estudiantes. REDU. Revista de Docencia Universitaria, 13(2), 363-390. https://doi.org/10.4995/redu.2015.5453

Alarcón, R., Blanca, M. J., & Bendayan, R. (2017). Student satisfaction with educational podcasts questionnaire. Escritos de Psicología-Psychological Writings, 10(2), 126-133. https://doi.org/10.5231/psy.writ.2017.14032.

Albaum, G. (1997). The Likert scale revisited. Market Research Society. Journal., 39(2), 1-21. https://doi.org/10.1177/147078539703900202.

Alles, M. A. (2002). Desempeño por competencias: evaluación de 360. Ediciones Granica SA.

Basurto-Mendoza, S. T., Cedeño, J. A. M., Espinales, A. N. V., & Gámez, M. R. (2021). Autoevaluación, Coevaluación y Heteroevaluación como enfoque innovador en la práctica pedagógica y su efecto en el proceso de enseñanza-aprendizaje. Polo del Conocimiento: Revista científico-profesional, 6(3), 828-845. https://doi.org/10.23857/pc.v6i3.2408

Baker, R. W., & Siryk, B. (1984). Student adaptation to college questionnaire (SACQ). Western Psychological Services. https://doi.org/10.1037/t06525-000.

Bisquerra Alzina, R., Martínez Olmo, F., Obiols Soler, M., & Pérez Escoda, N. (2006). Evaluación de 360º: Una aplicación a la educación emocional. Revista de investigación educativa, 24(1). https://revistas.um.es/rie/article/view/97371.

Blanco, A. (2008). Las rúbricas: un instrumento útil para la evaluación de competencias, In Prieto, L. (coord.). La enseñanza universitaria centrada en el aprendizaje: estrategias útiles para el profesorado (pp. 171-188). Octaedro-ICE de la Universidad de Barcelona.

Bravo Arteaga, A., & Fernández del Valle, J. C. (2000). La evaluación convencional frente a los nuevos modelos de evaluación auténtica. Psicothema, 12 (S. 2). http://hdl.handle.net/10651/27478

Boud, D., Cohen, R., & Sampson, J. (1999). Peer Learning and Assessment. Assessment & Evaluation in Higher Education, 24(4), 413–426. https://doi.org/10.1080/0260293990240405.

Casero-Martínez, A. (2008). Propuesta de un cuestionario de evaluación de la calidad docente universitaria consensuado entre alumnos y profesores. Revista de Investigación educativa, 26(1), 25-44. https://revistas.um.es/rie/article/view/94091

Castillo-Martín, C. (2021). Análisis de la eficiencia de los estudios de ingeniería. [Tesis Fin de Grado Ingeniería de Organización Industrial]. Universidad de Sevilla.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. https://doi.org/10.1007/BF02310555.

Douglas, J., Douglas, A., & Barnes, B. (2006). Measuring student satisfaction at a UK university. Quality assurance in education. https://doi.org/10.1108/09684880610678568.

Etxabe, J.M.; Aranguren, K. y Losada, D. (2011). Diseño de rúbricas en la formación inicial de maestros/as, Revista de Formación e Innovación Educativa Universitaria, 4 (3), 156-169.

García Cueto, E., Gallo Álvaro, P.M., & Miranda García, R. (1998). Bondad de ajuste en el análisis factorial confirmatorio. Psicothema, 10 (3). http://hdl.handle.net/10651/29218

García, M. E., Gutiérrez, A. B. B., & Rodríguez-Muñiz, L. J. (2016). Permanencia en la universidad: la importancia de un buen comienzo. Aula abierta, 44(1), 1-6. https://doi.org/10.1016/j.aula.2015.04.001

Glass, G. V., & Ellett Jr, F. S. (1980). Evaluation research. Annual review of psychology, 31(1), 211-228. https://doi.org/10.1146/annurev.ps.31.020180.001235

Gong, G. (2016). Consideration of evaluation of teaching at colleges. Open Journal of Social Sciences, 4(07), 82. http://dx.doi.org/10.4236/jss.2016.47013

González López, I., & López Cámara, A. B. (2010). Sentando las bases para la construcción de un modelo de evaluación a las competencias docentes del profesorado universitario. Revista de investigación educativa, 28(2), 403-423. https://revistas.um.es/rie/article/view/109431

González-Montesinos, M.J., & Backhoff, E. (2010). Validación de un cuestionario de contexto para evaluar sistemas educativos con modelos de ecuaciones estructurales. RELIEVE, 16(2), 1-17. https://doi.org/10.7203/relieve.16.2.4133

Jiménez Galán, Y.I., González Ramírez, M. A., & Hernández Jaime, J. (2010). Modelo 360 para la evaluación por competencias (enseñanza-aprendizaje). Innovación Educativa, 10(53), 43-53. https://www.redalyc.org/pdf/1794/179420770003.pdf.

Jöreskog, Karl G. (1994) On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59, 381-389. https://doi.org/10.1007/BF02296131

Hanrahan, S. J., & Isaacs, G. (2001). Assessing Self- and Peer-assessment: The students’ views. Higher Education Research & Development, 20(1), 53–70. https://doi.org/10.1080/07294360123776

Hernández, J., Pérez, J. A (dir.) (2019). La universidad española en cifras. Información académica, productiva y financiera de las Universidades Públicas de España. Indicadores Universitarios Curso académico 2017-2018. CRUE.

House, J. D. (1993). The relationship between academic self-concept and school withdrawal. The Journal of Social Psychology, 133(1), 125-127. https://doi.org/10.1080/00224545.1993.9712129

Lévy-Leboyer, C. (2004). Feedback de 360. Grupo Planeta (GBS).

Likert, R. (1932). A technique for the measurement of attitudes. Archives of psychology, 22 (140), 1-55.

López-Cózar-Navarro, C., Benito-Hernández, S., & Priede-Bergamini, T. (2020). Un análisis exploratorio de los factores que inciden en el abandono universitario en titulaciones de ingeniería. REDU. Revista de Docencia Universitaria, 18(2), 81-96. https://doi.org/10.4995/redu.2020.13294

López Pastor, V. M. L., Pascual, M. G., & Martín, J. B. (2005). La participación del alumnado en la evaluación: la autoevaluación, la coevaluación y la evaluación compartida. Revista Tándem: Didáctica de la Educación Física, 17, 21-37. http://hdl.handle.net/11162/21846

Martin, C. C., & Locke, K. D. (2022). What Do Peer Evaluations Represent? A Study of Rater Consensus and Target Personality. Frontiers in Education, 7. https://doi.org/10.3389/FEDUC.2022.746457

Martínez-Caro, E., & Campuzano-Bolarín, F. (2011). Factors affecting students’ satisfaction in engineering disciplines: traditional vs. blended approaches. European Journal of Engineering Education, 36(5), 473-483. https://doi.org/10.1080/03043797.2011.619647

Martínez-Figueira, E., Tellado-González, F., & Raposo-Rivas, M. (2013). La rúbrica como instrumento para la autoevaluación: un estudio piloto. REDU. Revista de Docencia Universitaria, 11(2), 373–390. https://doi.org/10.4995/redu.2013.5581

Mertler, Craig A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25). https://doi.org/10.7275/gcy8-0w24

Mehrens, W. A. (1992). Using performance assessment for accountability purposes. Educational Measurement: Issues and Practice. https://psycnet.apa.org/doi/10.1111/j.1745-3992.1992.tb00220.x

Ministerio de Universidades (2019). Datos y cifras del sistema universitario español. Publicación 2020-2021. Acceso el 01/07/2022.https://www.universidades.gob.es/stfls/universidades/Estadisticas/ficheros/Datos_y_Cifras_2020-21.pdf

Mumford, M. D., Baughman, W. A., Supinski, E. P., & Anderson, L. E. (1998). Assessing Complex Cognitive Shills. Beyond Multiple Choice: Evaluating Alternatives to Traditional Testing for Selection. Lawrence Erlbaum Associates, Pulishers.

Mumpuni, K. E., Priyayi, D. F., & Widoretno, S. (2022). How do Students Perform a Peer Assessment? International Journal of Instruction, 15(3), 751–766. https://doi.org/10.29333/IJI.2022.15341A

Nunnally, J.C., Bernstein, I. (1978) Psychometric Theory McGraw-Hill New York, the role of university in the development of entrepreneurial vocations: a Spanish study. The Journal of Technology Transfer, 37 (1), 387–405. https://doi.org/10.1007/s10961-010-9171-x.

Olds, B. M., & Miller, R. L. (2004). The effect of a first‐year integrated engineering curriculum on graduation rates and student satisfaction: A longitudinal study. Journal of Engineering Education, 93(1), 23-35. https://doi.org/10.1002/j.2168-9830.2004.tb00785.x

Orji, R., Nacke, L. E., & Di Marco, C. (2017, May). Towards personality-driven persuasive health games and gamified systems. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 1015-1027). https://doi.org/10.1145/3025453.3025577

Petkova, A. P., Domingo, M. A., & Lamm, E. (2021). Let’s be frank: Individual and team-level predictors of improvement in student teamwork effectiveness following peer-evaluation feedback. International Journal of Management Education, 19(3). https://doi.org/10.1016/J.IJME.2021.100538

Rodríguez González, R., Hernández García, J., Alonso Gutiérrez, A. M., & Díez Itza, E. (2003). El absentismo en la Universidad: resultados de una encuesta sobre motivos que señalan los estudiantes para no asistir a clase. Aula Abierta, 82.

Rodríguez Menéndez, M. D. C., Inda Caro, M. M., & Peña Calvo, J. V. (2015). Validación de la teoría cognitivo social de desarrollo de la carrera con una muestra de estudiantes de ingeniería. Educación XXI: revista de la Facultad de Educación, 18(2), 257-276. https://doi.org/10.5944/educxx1.14604

Rodríguez-Rodríguez, J., & Reguant-Álvarez, M. (2020). Calcular la fiabilidad de un cuestionario o escala mediante el SPSS: el coeficiente alfa de Cronbach. REIRE Revista d'Innovació i Recerca en Educació, 13(2), 1-13. https://doi.org/10.1344/reire2020.13.230048

Rojas-Torres, L. (2020). Robustez de los índices de ajuste del Análisis Factorial Confirmatorio a los valores extremos. Revista de Matemática: Teoría y Aplicaciones, 27(2), 403-424. https://doi.org/10.15517/rmta.v27i2.33677

Santos-Pastor, M.L., Cañadas, L., Martínez-Muñoz, L.F., & García-Rico, L. (2020). Diseño y validación de una escala para evaluar el aprendizaje-servicio universitario en actividad física y deporte. Educación XX1, 23(2), 67-93. https://doi.org/10.5944/educXX1.25422

Servimedia. (18 de diciembre de 2019). Los estudiantes de ciencias e ingenierías caen un 30% desde 2.000 porque el mercado laboral no recompensa "el esfuerzo". El Economista. https://www.eleconomista.es/ecoaula/noticias/10260885/12/19/Los-estudiantes-de-ciencias-e-ingenierias-caen-un-30-desde-2000-porque-el-mercado-laboral-no-recompensa-el-esfuerzo.html

Silió, E. (18 de diciembre de 2019). La revolución 4.0 peligra: los estudiantes de ingeniería caen un 30% en 20 años. El País. https://elpais.com/sociedad/2019/12/17/actualidad/1576612459_205974.html

Sotelo, A. F., & Arévalo, M. G. V. (2015). Proceso de autoevaluación, coevaluación y heteroevaluación para caracterizar el comportamiento estudiantil y mejorar su desempeño. Revista San Gregorio, 1(9), 6-15.

Schertzer, C. B., & Schertzer, S. M. (2004). Student satisfaction and retention: A conceptual model. Journal of Marketing for Higher Education, 14(1), 79-91. https://doi.org/10.1300/J050v14n01_05

Stegmann, J.G. (18 de diciembre de 2019). En España nadie quiere estudiar ingenierías y el país se dirige a la «dependencia tecnológica». ABC. https://www.abc.es/sociedad/abci-espana-nadie-quiere-estudiar-ingenierias-y-pais-dirige-dependencia-tecnologica-201912181146_noticia.html

Tuan, H. L., Chin, C. C., & Shieh, S. H. (2005). The development of a questionnaire to measure students' motivation towards science learning. International journal of science education, 27(6), 639-654. https://doi.org/10.1080/0950069042000323737

Tyler, R. W. (1942). General statement on evaluation. The Journal of Educational Research, 35(7), 492-501. https://doi.org/10.1080/00220671.1942.10881106

Topping, K. (1998). Peer assessment between students in colleges and universities. Review of educational Research, 68(3), 249-276. https://doi.org/10.3102/00346543068003249

Vivanco-Álvarez, R. V., & Pinto-Vilca, S. A. (2018). Efecto de la aplicación de coevaluación sobre la motivación de logro en estudiantes de nivel universitario. Paideia XXI, 8(1), 57-78. https://doi.org/10.31381/paideia.v8i1.2037

Wiggins, G. (1990). The Case for Authentic Assessment. ERIC Digest.

Wortman, P. M. (1983). Evaluation research: A methodological perspective. Annual Review of Psychology, 34(1), 223-260. https://doi.org/10.1146/annurev.ps.34.020183.001255

Yani, B., Harding, A., & Engelbrecht, J. (2019). Academic maturity of students in an extended programme in mathematics. International journal of mathematical education in science and technology, 50(7), 1037-1049. https://doi.org/10.1080/0020739X.2019.1650305