Revisitando las revisiones sistemáticas desde la perspectiva metodológica

M. Teresa Anguera

doi:10.30827/relieve.v29i1.27758

Authors

M. Teresa Anguera Universidad de Barcelona https://orcid.org/0000-0001-7147-2927

DOI:

https://doi.org/10.30827/relieve.v29i1.27758

Keywords:

Síntesis, mixed methods, revisiones agregativas y configurativas, calidad metodológica, diagrama PRISMA

Abstract

The rapid expansion of systematic reviews in recent years has led to growth that has often lacked the required oversight. A large number of systematic reviews have been performed in practically all contexts and sub-contexts of knowledge. This has influenced the movement towards evidence-based practice and ease of access to a wide array of bibliographic sources. Nonetheless, this also makes it necessary to reposition systematic reviews from a methodological perspective. Alongside this, for various reasons, such as being alert to the often-incorrect use of terminology and procedural concepts, it will be necessary to redefine their aims and approach regarding strategy, propose an appropriate structure, bestow them with optimal breadth, depth and coverage, and evaluate the methodological quality of the primary documents on which they are based. In the present context, it is urged to structure such reviews from a mixed methods approach, which offers a continuum between the qualitative at one end and the quantitative at the other.

Downloads

Download data is not yet available.

Author Biography

M. Teresa Anguera, Universidad de Barcelona

Professor of Methodology of Behavioral Sciences at the Faculty of Psychology of the University of Barcelona (Spain). She has a degree in Psychology and Law, and a PhD in Psychology. His main lines of research are: Observational methodology, mixed methods, and evaluation of low intervention programs. He has participated in 30 competitive research projects, having coordinated 12 of them at a national level. He has made numerous publications in Spain and abroad (31 books, 139 book chapters, and 331 articles). He has directed/co-directed 66 Doctoral Theses already defended. She is a member of the Spanish Academy of Psychology, the Royal European Academy of Doctors, and a corresponding member of the Royal Academy of Medicine of Catalonia.

References

Abrami, P.C., Borokhovski, E., Bernard, R.M., Wade, C.A., Tamim, R., Persson, T., & Surkes, M.A. (2010). Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice, 6, 371–389. https://doi.org/10.1332/174426410X524866

Ailinger, R. (2003). Contribution of qualitative evidence to evidence based practice in nursing. Revista Latinoamericana de Enfermagem, 11(3), 275-279. https://doi.org/10.1590/S0104-11692003000300002

Albrecht, B.M., Foettinger, L., & Bammann, K. (2021). Men’s sheds as community based health promotion for men aged 50 plus: protocol for a mixed-methods systematic review. Systematic Reviews, 10, 215. https://doi.org/10.1186/s13643-021-01762-x

Allers, K., Hoffmann, F., Mathes, T., & Pieper, D. (2018). Systematic reviews with published protocols compared to those without: more effort, older search. Journal of Clinical Epidemiology, 95,102–10. https://doi.org/10.1016/j.jclinepi.2017.12.005

Allwood, C.M. (2012). The distinction between qualitative and quantitative research methods is problematic. Quality & Quantity, 46, 1417-1429. https://doi.org/10.1007/s11135-011-9455-8

Anguera, M.T. (2022). Profundizando en el análisis en mixed methods: Integración de elementos cualitativos y cuantitativos en el marco de la observación sistemática del comportamiento. Discurso Doctorado Honoris Causa. Tenerife: Universidad de La Laguna.

Anguera, M.T., Blanco-Villaseñor, A., Losada, J.L., Sánchez-Algarra, P., & Onwuegbuzie, A.J. (2018). Revisiting the difference between mixed methods and multimethods: Is it all in the name? Quality & Quantity, 52, 2757-2770. https://doi.org/10.1007/s11135-018-0700-2

Anguera, M.T., Jonsson, G.K., Escolano-Pérez, E., Sánchez-López, C.R., Losada, J.L., & Portell, M. (2023). T-pattern detection in the scientific literature of this century: A systematic review. Frontiers in Psychology, 14,1085980. https://doi.org/10.3389/fpsyg.2023.1085980

Aromataris, E., & Munn, Z. (Eds.) (2021). JBI Manual for Evidence Synthesis. https://doi.org/10.46658/JBIMES-20-01

Backman, C., Crick, M., Cho-Young, D., Scharf, M., & Shea, B. (2018). What is the impact of sensory practices on the quality of life of long-term care residents? A mixed-methods systematic review protocol. Systematic Reviews, 7, 115. https://doi.org/10.1186/s13643-018-0783-9

Barbour, R.S., & Barbour, M. (2003). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9, 179-186. https://doi.org/10.1046/j.1365-2753.2003.00371.x

Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9,59.

Boaz, A., Ashby, D., Denyer, D., Egan, M., Harden, A., Jones, D. R., . . ., & Tranfield, D. (2006). A multitude of syntheses: A comparison of five approaches from diverse policy fields. Evidence & Policy, 2, 479-502. https://doi.org/10.1332/174426406778881755

Bohlin, I. (2012). Formalizing syntheses of medical knowledge: The rise of meta-analysis and systematic reviews. Perspectives on Science, 20(3), 273-309. https://doi.org/10.1162/POSC_a_00075

Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011). An international registry of systematic-review protocols. Lancet, 377, 108-109. https://doi.org/10.1016/S0140-6736(10)60903-8

Brignardello-Petersen, R., Ioannidis, J.P.A., Tomlimson, G., & Guyatt, G. (2015). Surprising results of randomized trials. In G. Guyatt, D. Rennie, M.O. Meade, and D.J. Cook (Eds.), Users’ guide to the medical literature. A manual for evidence-based clinical practice, 3rd ed. McGraw-Hill.

Broome, M.E. (1993). Integrative literature reviews for the development of concepts. In B.L. Rodgers, and K.A. Knafl (Eds.), Concept development in nursing (pp. 231-250). W.B. Saunders Co.

Brown, J.V.E., Ajjan, R., Siddiqi, N., & Coventry, P.A. (2022). Acceptability and feasibility of continuous glucosa monitoring in people with diabetes: protocol for a mixed-methods systematic review of quanitative and qualitative evidence. Systematic Reviews, 11, 263. https://doi.org/10.1186/s13643-022-02126-9

Bunn, F., Trivedi, D., Alderson, P., Hamilton, L., Martin, A., Pinkney, E., & Iliffe, S. (2015). The impact of Cochrane Reviews: A mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research. Health Technology Assessment, 19(28), 1-100. https://doi.org/10.3310/hta19280

Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. https://doi.org/10.1186/1748-5908-2-40

Collins, K. & O’Cathain, A. (2009). Ten points about mixed methods research to be considered by the novice researcher. International Journal of Multiple Research Approaches, 3, 2-7. https://doi.org/10.5172/mra.455.3.1.2

Chacón-Moscoso, S., Sanduvete-Chaves, S., Lozano-Lozano, J.A., Portell, M., & Anguera, M.T. (2021). From randomized control trial to mixed methods: A practical framework for program evaluation based on methodological quality. Anales de Psicología / Annals of Psychology 37(3), 599-608. https://doi.org/10.6018/analesps.470021

Chalmers, I., Hedges, L., & Cooper, H. (2002). A brief history of research synthesis. Evaluation of Health Professions, 25, 12–37. https://doi.org/10.1177/0163278702025001003

Chang, Y., Voils, C. I., Sandelowski, M., & Crandell, J. L. (2009). Transforming verbal counts in reports of qualitative descriptive studies into numbers. Western Journal of Nursing Research, 31(7), 837–852. https://doi.org/10.1177/0193945909334434

Chicoine, G., Côté, J., Pepin, J., Fontaine, G., Maheu-Cadotte, M.-A., Hong, Q.N., Rouleau, G., Ziegler, D., & Jutras-Aswad, D. (2021). Effectiveness and experiencies of the Extension for Community Healthcare Outcomes (ECHO) Model in developing competencies among healthcare professionals: a mixed methods systematic review protocol. Systematic Reviews, 10, 313. https://doi.org/10.1186/s13643-021-01832-0

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesizing qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy, 10, 45-53. https://doi.org/10.1258/1355819052801804

Eisner, E.W. (1991). The enlightened eye: Qualitative inquiry and the enhancement of educational practice. MacMillan.

Fàbregues, S., Molina-Azorín, J.F., & Fetters, M.D. (2021). Virtual special issue on “Quality in Mixed Methods Research”. Journal of Mixed Methods Research, 15(2), 146-151. https://doi.org/10.1177/15586898211001974

Fàbregues, S., Paré, M. H., & Meneses, J. (2019). Operationalizing and conceptualizing quality in mixed methods research: A multiple case study of the disciplines of education, nursing, psychology, and sociology. Journal of Mixed Methods Research, 13(4), 424-445. https://doi.org/10.1177/155868981 7751774

Glasziou, P.P., & Sanders, S.L. (2002). Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine, 21, 1503-1511. https://doi.org/10.1002/sim.1183

Greckhamer, T., & Koro-Ljungberg, M. (2005). The erosion of a method: Examples from grounded theory. International Journal of Qualitative Studies in Education, 18, 729-750. https://doi.org/10.1080/09518390500298204

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Social Science Medicine, 61,417–430. https://doi.org/10.1016/j.socscimed.2004.12.001

Gough, D. (2007, January). Dimensions of difference in evidence reviews. National Centre for Research Methods Meeting. Manchester: EPPI-Centre. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1919

Gough, D., & Thomas, J. (2012). Commonality and diversity in reviews. In D. Gough, S. Oliver, & J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 35-65). Sage.

Gough, D. (2015). Qualitative and mixed methods in systematic reviews. Systematic Reviews, 4, 181. https://doi.org/10.1186/s13643-015-0151-y

Gough, D., Oliver, S., Newman, M., & Bird, K. (2009). Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. London: Teaching and Learning Research Programme.

Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1, 28. https://doi.org/10.1186/2046-4053-1-28

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine, 61, 417-430. https://doi.org/10.1016/j.socscimed.2004.12.001

Harden, A., & Gough, D. (2012). Quality and relevance appraisal. In D. Gough, S. Oliver and J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 153-178). Sage.

Harden, A., & Thomas, J. (2005). Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methodology, 8, 257-271. https://doi.org/10.1080/13645570500155078

Hawker, S., Payne, S., Kerr, C., Hardey, M., & Powell, J. (2002). Appraising the evidence: Reviewing the disparate data systematically. Qualitative Health Research, 12, 1284-1299. https://doi.org/10.1177/1049732302238251

Heyvaert, M., Hannes, K., & Onghena, P. (2016). Using mixed methods research synthesis for literature reviews: The mixed methods research synthesis approach. Sage. https://doi.org/10.4135/9781506333243

Heyvaert, M., Maes, B., & Onghena, P. (2013). Mixed methods research synthesis: Definition, framework, and potential. Quality & Quantity, 47, 659-676. https://doi.org/10.1007/s11135-011-9538-6

Hoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., et al. (2021). Nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. Journal of Clinical Epidemiology, 138, 1–11. https://doi.org/10.1016/j.jclinepi.2021.05.022

Hong, Q.M., & Pluye, P. (2019). A conceptual framework for critical appraisal in systrematic mixed studies reviews. Journal of Mixed Methods Research, 13(4), 446-460. https://doi.org/10.1177/1558689818770058

Hong, Q.M., Pluye, P., Bujold, M., & Wassef, M. (2017). Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Systematic Reviews, 6:61. https://doi.org/10.1186/s13643-017-0454-2

JBI Manual for Evidence Synthesis (2021). CRICOS Provider Number 00123M. jbi.global

Johnson, E.E., O’Keefe, H., Sutton, A., & Marshall, C. (2022). The Systematic Review Toolbox: keeping up to date with tools to support evidence sythesis. Systematic Reviews, 11, 258. https://doi.org/10.1186/s13643-022-02122-z

Johnson, R.B., & Onwuegbuzie, A.J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33, 14-26. https://doi.org/10.3102/0013189X033007014

Johnson, R.B., Onwuegbuzie, A.J., & Turner, L.A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–33. https://doi.org/10.1177/1558689806298224

Johnstone, A., McCrorie, P., Cordovil, R., Fjørtoft, I., Livonen, S., Jidovtseff, B., Lopes, F., Reilly, J.J., Thomson, H., Wells, V., & Martin, A. (2020). Nature-based early childhood education for child health, wellbeing and development: a mixed-methods systematic review protocol. Systematic Reviews, 9, 226. https://doi.org/10.1186/s13643-020-01489-1

Jones, K. (2004). Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. The Qualitative Report, 9(1), 95-112.

Kirkham, J.J., Altman, D.G., & Williamson, P.R. (2010). Bias due to changes in specified outcomes during the systematic review process. PLoS One, 5, e9810. https://doi.org/10.1371/journal.pone.0009810

Krleza-Jeric, K., Chan, A.W., Dickersin, K., Sim, I., Grimshaw, J., & Gluud, C. (2005). Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ, 330(7497), 956–8. https://doi.org/10.1136/bmj.330.7497.956

Leslie, L., Cherry, R.F., Mulla, A., Abbott, J., Furfari, K., Glover, J.J., Hamke, B., Wynia, M.K. (2016). Domains of quality for clinical ethics case consultation: a mixed-method systematic review. Systematic Reviews, 5, 95. https://doi.org/10.1186/s13643-016-0273-x

Lockwood, C., Porrit, K., Munn, Z., Rittenmeyer, L., Salmond, S., Bjerrum, M., Loveday, H., Carrier, J, & Stannard, D. (2020). Systematic reviews of qualitative evidence. In E. Aromataris, and Z. Munn (Eds.), JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/JBIMES-20-03

Lucas, P.J., Baird, J., Arai, L., Law, C., & Roberts, H.M. (2007). Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Medical Research Methodology, 7, 4. Retrieved from http://www.biomedcentral.com/1471-2288/7/4

Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research and Policy, 10 (Suppl. 1), 6-20. https://doi.org/10.1258/1355819054308576

Maxwell, J.A. (2004a). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33, 3-11. https://doi.org/10.3102/0013189X033002003

Maxwell, J.A. (2004b). Using qualitative methods for causal explanation. Field Methods, 16, 243-264. https://doi.org/10.1177/1525822X04266831

Mazevska, D., Pearse, J, & Tierney, S. (2022). Using a theoretical framework to inform implementation of the patient-centred medical home (PCMH) model in primary care: protocol for a mixed-methods systematic review. Systematic Reviews, 11, 249. https://doi.org/10.1186/s13643-022-02132-x

Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4,1. https://doi.org/10.1186/2046-4053-4-1

Morse, J.M. (1994). On the crest of a wave? (Editorial). Qualitative Health Research, 4, 139-141. https://doi.org/10.1177/104973239400400201

Munafo, M.R., Hollands, G.J., & Marteau, T.M. (2018). Open science prevents mindless science. BMJ, 363, k4309. https://doi.org/10.1136/bmj.k4309

Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Southern Illinois University Press.

Newman, I., & Hitchcock, J. H. (2011). Underlying agreements between quantitative and qualitative research: The short and tall of it all. Human Resource Development Review, 10(4), 381-398. https://doi.org/10.1177/1534484311413867

Onwuegbuzie, A. (2012). Putting the MIXED back into quantitative and qualitative research in educational reearch and beyond: Moving towards the ‘radical middle’. International Journal of Multiple Research Approaches, 6, 192-219. https://doi.org/10.5172/mra.2012.6.3.192

Onwuegbuzie, A.J. & Teddie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori & C. Teddie (Edss), Handbook of mixed metods in social and behavioral research (pp. 351-383). Sage.

Onwuegbuzie, A., & Tashakkori, A. (2015). Utilizing mixed research and evaluation methodology in peace psychology and beyond. In D. Bretherton & S. F. Law (Eds.), Methodologies in peace psychology: Peace research by peaceful means (pp. 115-137). New York, NY: Springer. https://doi.org/10.1007/978-3-319-18395-4_6

Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44(3), 795–805. https://doi.org/10.3758/s13428-011-0180-y

Pace, R., Pluye, P., Bartlett, G., Macaulay, A.C., Salsberg, J., Jagosh, J., et al. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. International Journal of Nursing Studies, 49 (1), 47–53. https://doi.org/10.1016/j.ijnurstu.2011.07.002

Page, M.J., Shamseer, L., & Tricco, A.C. (2018). Registration of systematic reviews in PROSPERO: 30,000 records and counting. Systematic Reviews, 7(1), 32. https://doi.org/10.1186/s13643-018-0699-4

Parkhurst, P.E., Lovell, K.L., Sprafka, S.A., & Hodgins, M. (1972). Evaluation of videodisc modules: A mixed methods approach. (ERIC Document Reproduction Service No. ED348014)

Pawson, R. (2006). Evidenced-based policy: A realist perspective. Sage. https://doi.org/10.4135/9781849209120

Pearson, S.A., Taylor, S, Marsden, A., & Yorke, J. (2021). Access to systemic anti-cancer therapies for women with secondary breast cancer – protocol for a mixed methods systematic review. Systematic Reviews, 10, 209. https://doi.org/10.1186/s13643-021-01761-y

PLoS Medicine Editors (2011). Best Practice in Systematic Reviews: The Importance of Protocols and Registration. PLoS Medicine, 8(2), e1001009. https://doi.org/10.1371/journal.pmed.1001009

Pluye, P. (2015). Mixed kinds of evidence: synthesis designs and critical appraisal for systematic mixed studies reviews including qualitative, quantitative and mixed methods studies. Evidence Based Medicine. Letter. https://doi.org/10.1136/ebmed-2014-110158

Pluye, P., Gagnon, M.P., Griffiths, F., Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative, and mixed methods primary-level studies in mixed studies reviews. International Journal of Nursing Studies, 46, 529-546. https://doi.org/10.1016/j.ijnurstu.2009.01.009

Pluye, P., & Hong, Q. N. (2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35, 29-45. https://doi.org/10.1146/annurev-publhealth-032013-182440

Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., et al. (2011). Proposal: A Mixed Methods Appraisal Tool for Systematic Mixed Studies Reviews. http://mixedmethodsapprai-saltoolpublic.pbworks.com

Pound, P., Britten, N., Morgan, M., Yardley, L., Pope, C., Daker-White, G., et al. (2005), Resisting medicines: A sythesis of qualitative studies of medicine taking. Social Science & Medicine, 6, 133-155. https://doi.org/10.1016/j.socscimed.2004.11.063

Preciado, M., Anguera, M.T., Olarte, M., & Lapresa, D. (2019). Observational studies in male elite football: A systematic mixed study review. Frontiers in Psychology, 10, 2077. https://doi.org/10.3389/fpsyg.2019.02077

Preciado, M., Anguera, M.T., Olarte, M., y Lapresa, D. (2021). Revisión Sistemática en Fútbol Sala desde los Mixed Methods. Revista de Psicología del Deporte / Journal of Sports Psychology, 30(1), 75-96.

Queiroga Souto, R., Khanassov, V., Hong, Q.N., Busch, P.L., Vedel, Is., & Pluye, P. (2015). Systematic mixed studies reviews: Updating results on the reliability and efficiency of the mixed methods appraisal tool. International Journal of Nursing Studies, 52, 500-501. https://doi.org/10.1016/j.ijnurstu.2014.08.010

Rana, D., Westrop, S., Germeni, E., McGarty, A., Ells, L., Lally, P., McEwan, M., Melville, C., Harris, L., & Wu, O. (2021). Understanding the effectiveness and underlying mechanisms of lifestyle modification interventions in adults with learning disabilities: protocol for a mixed-methods systematic review. Systematic Reviews, 10, 251. https://doi.org/10.1186/s13643-021-01808-0

Ridenour, C.S., & Newman, I. (2008). Mixed methods research: Exploring the interactive continuum. Southern Illinois University Press.

Sandelowski, M. (2004). Using qualitative research. Qualitative Health Research, 14, 1366-1386. https://doi.org/10.1177/1049732304269672

Sandelowski, M. (2014). Unmixing mixed-methods research. Research in Nursing & Health, 37, 3–8. https://doi.org/10.1002/nur.21570

Sandelowski, M., & Barroso, J. (2007). Handbook for synthesizing qualitative research. Springer.

Sandelowski, M, Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208-222. https://doi.org/10.1177/1558689809334210

Sandelowski, M., Voils, C.I., & Barroso, J. (2006). Defining and designing mixed research synthesis studies. Research in the Schools, 13, 29-40.

Sandelowski, M., Voils, C.J., Leeman, J., & Crandlee, J.L. (2011). Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research, 6(4), 317-331. https://doi.org/10.1177/1558689811427913.

Sarmento, H., Anguera, M.T., Pereira, A., & Araujo, D. (2018). Talent Identification and Development in Male Football: A Systematic Review. Sports Medicine, 48(4), 907-931. https://doi.org/10.1007/s40279-017-0851-7

Silagy, C.A., Middleton, P., & Hopewell, S. (2002). Publishing protocols of systematic reviews: Comparing what was done to what was planned. JAMA, 287, 2831-2834. https://doi.org/10.1001/jama.287.21.2831

Smith, V., Devane, D., Begley, C.M., & Clarke, M. (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11, 15. https://doi.org/10.1186/1471-2288-11-15

Souto, R., Khanassov, V., Pluye, P., Hong, Q.N., Bush, P., Vedel, I. (2014). Systematic mixed studies reviews: reliability testing of the Mixed Methods Appraisal Tool. In: Mixed Methods International Research Association (MMIRA) Conference. Boston, USA. http:// mixedmethodsappraisaltoolpublic.pbworks.com

Stewart, R., & Oliver, S. (2012). Making a difference with systematic reviews. In D. Gough, S. Oliver, and J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 227-244). Sage.

Straus, S., & Moher, D. (2010). Registering systematic reviews. CMAJ, 182, 13-14. https://doi.org/10.1186/1471-2288-11-15

Suri, H., & Clarke, D. (2009). Advancements in research synthesis methods: From a methodologically inclusive perspective. Review of Educational Research, 79(1), 395-430. https://doi.org/10.3102/0034654308326349

Sutton, A.J., Abrams, K.R., Jones, D.R., Sheldon, T.A., & Song, F. (2000). Methods for meta-analysis in medical research. Wiley.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010). Handbook of mixed methods in social and behavioral research (2nd ed.). Sage.

Taylor, C., & Coffey, A. (2009). Special issue: Qualitative research and methodological innovation. Qualitative Research, 9, 523-526. https://doi.org/10.1177/1468794109350355

Teddlie, C. & Tashakkori, A. (2006). A general typology of research designs featuring mixed methods. Research in the Schools, 13, 12-28.

Tennant, J.P., Waldner, F., Jacques, D.C., Masuzzo, P., Collister, L.B., Hartgerink, C.H. (2016). The academic, economic and societal impacts of open access: an evidence-based review. F1000Research, 5, 632. https://doi.org/10.12688/f1000research.8460.3.

Travers, M. (2009). New methods, old problems: A skeptical view of innovation in qualitative research. Qualitative Research, 9, 161-179. https://doi.org/10.1177/1468794108095079

Tricco, A.C., Pham, B., Brehaut, J., Tetroe, J., & Cappelli, M. (2009). An international survey indicated that unpublihed systematic reviews exist. Journal of Clinical Epidemiology, 62, 617-623. https://doi.org/10.1016/j.jclinepi.2008.09.014

Trinder, L. (2000). A critical appraisal of evidence-based practice. In L. Trinder & S. Reynols (Eds.), Evidence-based practice: A critical appraisal (pp. 212-241). Bkackwell Science. https://doi.org/10.1002/9780470699003

Trinder, L. & Reynolds, S. (Eds.) (2000). Evidence-based practice: A critical appraisal. Blackwell Science. https://doi.org/10.1002/9780470699003.ch10

Tronchoni, H., Izquierdo, C., & Anguera, M.T. (2022). A systematic review on lecturing in contemporary university teaching. Frontiers in. Psychology, 13, 971617. https://doi.org/10.3389/fpsyg.2022.971617

Tufanaru, C., Munn, Z., Aromataris, E., Campbell, J., & Hopp, L. (2020). Systematic reviews of effectiveness. In E. Aromataris, and Z. Munn (Eds.), JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/JBIMES-20-04

Van der Braak, K., Ghannad, M., Orelio, C., Heus, P., Damen, J.A.A, Spijker, R., Robinson, K., Lund, H., & Hooft, L. (2022). The score after 10 years of registration of systematic review protocols. Systematic Reviews, 11, 191. https://doi.org/10.1186/s13643-022-02053-9

Vandyk, A.D., Kaluzienski, M., Goldie, C., Stokes, Y., Ross-White, A., Kronick, J., Gilmour, M., MacPhee, C., & Graham, I.D. (2019). Interventions to improve emergency department use for mental health reasons: protocol for a mixed-methods systematic review. Systematic Reviews, 8, 84. https://doi.org/10.1186/s13643-019-1008-6

Vicente-Sáezooft, L. (2022), R., & Martínez-Fuentes, C. (2018). Open science now: a systematic literature review for an integrated definition. Journal of Business Research, 88, 428–36. https://doi.org/10.1016/j.jbusres.2017.12.043

Voils, C.I., Sandelowski, M., Barroso, J., Hasselblad, V. (2008). Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods, 20(1), 3-25. https://doi.org/10.1177/1525822X07307463

Wiafe, E., Mensah, K.B., Mensah, A.B.B., Bandalee, V., & Oosthuizen, F. (2020). The awareness of women on prostate cancer: a mixed-methods systematic review protocol. Systematic Reviews, 9, 253. https://doi.org/10.1186/s13643-020-01513-4