Revisiting systematic reviews from a methodological perspective
DOI:
https://doi.org/10.30827/relieve.v29i1.27758Keywords:
Síntesis, mixed methods, revisiones agregativas y configurativas, calidad metodológica, diagrama PRISMAAbstract
The rapid expansion of systematic reviews in recent years has led to growth that has often lacked the required oversight. A large number of systematic reviews have been performed in practically all contexts and sub-contexts of knowledge. This has influenced the movement towards evidence-based practice and ease of access to a wide array of bibliographic sources. Nonetheless, this also makes it necessary to reposition systematic reviews from a methodological perspective. Alongside this, for various reasons, such as being alert to the often-incorrect use of terminology and procedural concepts, it will be necessary to redefine their aims and approach regarding strategy, propose an appropriate structure, bestow them with optimal breadth, depth and coverage, and evaluate the methodological quality of the primary documents on which they are based. In the present context, it is urged to structure such reviews from a mixed methods approach, which offers a continuum between the qualitative at one end and the quantitative at the other.
Downloads
References
Abrami, P.C., Borokhovski, E., Bernard, R.M., Wade, C.A., Tamim, R., Persson, T., & Surkes, M.A. (2010). Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice, 6, 371–389. https://doi.org/10.1332/174426410X524866
Ailinger, R. (2003). Contribution of qualitative evidence to evidence based practice in nursing. Revista Latinoamericana de Enfermagem, 11(3), 275-279. https://doi.org/10.1590/S0104-11692003000300002
Albrecht, B.M., Foettinger, L., & Bammann, K. (2021). Men’s sheds as community based health promotion for men aged 50 plus: protocol for a mixed-methods systematic review. Systematic Reviews, 10, 215. https://doi.org/10.1186/s13643-021-01762-x
Allers, K., Hoffmann, F., Mathes, T., & Pieper, D. (2018). Systematic reviews with published protocols compared to those without: more effort, older search. Journal of Clinical Epidemiology, 95,102–10. https://doi.org/10.1016/j.jclinepi.2017.12.005
Allwood, C.M. (2012). The distinction between qualitative and quantitative research methods is problematic. Quality & Quantity, 46, 1417-1429. https://doi.org/10.1007/s11135-011-9455-8
Anguera, M.T. (2022). Profundizando en el análisis en mixed methods: Integración de elementos cualitativos y cuantitativos en el marco de la observación sistemática del comportamiento. Discurso Doctorado Honoris Causa. Tenerife: Universidad de La Laguna.
Anguera, M.T., Blanco-Villaseñor, A., Losada, J.L., Sánchez-Algarra, P., & Onwuegbuzie, A.J. (2018). Revisiting the difference between mixed methods and multimethods: Is it all in the name? Quality & Quantity, 52, 2757-2770. https://doi.org/10.1007/s11135-018-0700-2
Anguera, M.T., Jonsson, G.K., Escolano-Pérez, E., Sánchez-López, C.R., Losada, J.L., & Portell, M. (2023). T-pattern detection in the scientific literature of this century: A systematic review. Frontiers in Psychology, 14,1085980. https://doi.org/10.3389/fpsyg.2023.1085980
Aromataris, E., & Munn, Z. (Eds.) (2021). JBI Manual for Evidence Synthesis. https://doi.org/10.46658/JBIMES-20-01
Backman, C., Crick, M., Cho-Young, D., Scharf, M., & Shea, B. (2018). What is the impact of sensory practices on the quality of life of long-term care residents? A mixed-methods systematic review protocol. Systematic Reviews, 7, 115. https://doi.org/10.1186/s13643-018-0783-9
Barbour, R.S., & Barbour, M. (2003). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9, 179-186. https://doi.org/10.1046/j.1365-2753.2003.00371.x
Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9,59.
Boaz, A., Ashby, D., Denyer, D., Egan, M., Harden, A., Jones, D. R., . . ., & Tranfield, D. (2006). A multitude of syntheses: A comparison of five approaches from diverse policy fields. Evidence & Policy, 2, 479-502. https://doi.org/10.1332/174426406778881755
Bohlin, I. (2012). Formalizing syntheses of medical knowledge: The rise of meta-analysis and systematic reviews. Perspectives on Science, 20(3), 273-309. https://doi.org/10.1162/POSC_a_00075
Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011). An international registry of systematic-review protocols. Lancet, 377, 108-109. https://doi.org/10.1016/S0140-6736(10)60903-8
Brignardello-Petersen, R., Ioannidis, J.P.A., Tomlimson, G., & Guyatt, G. (2015). Surprising results of randomized trials. In G. Guyatt, D. Rennie, M.O. Meade, and D.J. Cook (Eds.), Users’ guide to the medical literature. A manual for evidence-based clinical practice, 3rd ed. McGraw-Hill.
Broome, M.E. (1993). Integrative literature reviews for the development of concepts. In B.L. Rodgers, and K.A. Knafl (Eds.), Concept development in nursing (pp. 231-250). W.B. Saunders Co.
Brown, J.V.E., Ajjan, R., Siddiqi, N., & Coventry, P.A. (2022). Acceptability and feasibility of continuous glucosa monitoring in people with diabetes: protocol for a mixed-methods systematic review of quanitative and qualitative evidence. Systematic Reviews, 11, 263. https://doi.org/10.1186/s13643-022-02126-9
Bunn, F., Trivedi, D., Alderson, P., Hamilton, L., Martin, A., Pinkney, E., & Iliffe, S. (2015). The impact of Cochrane Reviews: A mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research. Health Technology Assessment, 19(28), 1-100. https://doi.org/10.3310/hta19280
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. https://doi.org/10.1186/1748-5908-2-40
Collins, K. & O’Cathain, A. (2009). Ten points about mixed methods research to be considered by the novice researcher. International Journal of Multiple Research Approaches, 3, 2-7. https://doi.org/10.5172/mra.455.3.1.2
Chacón-Moscoso, S., Sanduvete-Chaves, S., Lozano-Lozano, J.A., Portell, M., & Anguera, M.T. (2021). From randomized control trial to mixed methods: A practical framework for program evaluation based on methodological quality. Anales de Psicología / Annals of Psychology 37(3), 599-608. https://doi.org/10.6018/analesps.470021
Chalmers, I., Hedges, L., & Cooper, H. (2002). A brief history of research synthesis. Evaluation of Health Professions, 25, 12–37. https://doi.org/10.1177/0163278702025001003
Chang, Y., Voils, C. I., Sandelowski, M., & Crandell, J. L. (2009). Transforming verbal counts in reports of qualitative descriptive studies into numbers. Western Journal of Nursing Research, 31(7), 837–852. https://doi.org/10.1177/0193945909334434
Chicoine, G., Côté, J., Pepin, J., Fontaine, G., Maheu-Cadotte, M.-A., Hong, Q.N., Rouleau, G., Ziegler, D., & Jutras-Aswad, D. (2021). Effectiveness and experiencies of the Extension for Community Healthcare Outcomes (ECHO) Model in developing competencies among healthcare professionals: a mixed methods systematic review protocol. Systematic Reviews, 10, 313. https://doi.org/10.1186/s13643-021-01832-0
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesizing qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy, 10, 45-53. https://doi.org/10.1258/1355819052801804
Eisner, E.W. (1991). The enlightened eye: Qualitative inquiry and the enhancement of educational practice. MacMillan.
Fàbregues, S., Molina-Azorín, J.F., & Fetters, M.D. (2021). Virtual special issue on “Quality in Mixed Methods Research”. Journal of Mixed Methods Research, 15(2), 146-151. https://doi.org/10.1177/15586898211001974
Fàbregues, S., Paré, M. H., & Meneses, J. (2019). Operationalizing and conceptualizing quality in mixed methods research: A multiple case study of the disciplines of education, nursing, psychology, and sociology. Journal of Mixed Methods Research, 13(4), 424-445. https://doi.org/10.1177/155868981 7751774
Glasziou, P.P., & Sanders, S.L. (2002). Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine, 21, 1503-1511. https://doi.org/10.1002/sim.1183
Greckhamer, T., & Koro-Ljungberg, M. (2005). The erosion of a method: Examples from grounded theory. International Journal of Qualitative Studies in Education, 18, 729-750. https://doi.org/10.1080/09518390500298204
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Social Science Medicine, 61,417–430. https://doi.org/10.1016/j.socscimed.2004.12.001
Gough, D. (2007, January). Dimensions of difference in evidence reviews. National Centre for Research Methods Meeting. Manchester: EPPI-Centre. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1919
Gough, D., & Thomas, J. (2012). Commonality and diversity in reviews. In D. Gough, S. Oliver, & J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 35-65). Sage.
Gough, D. (2015). Qualitative and mixed methods in systematic reviews. Systematic Reviews, 4, 181. https://doi.org/10.1186/s13643-015-0151-y
Gough, D., Oliver, S., Newman, M., & Bird, K. (2009). Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. London: Teaching and Learning Research Programme.
Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1, 28. https://doi.org/10.1186/2046-4053-1-28
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine, 61, 417-430. https://doi.org/10.1016/j.socscimed.2004.12.001
Harden, A., & Gough, D. (2012). Quality and relevance appraisal. In D. Gough, S. Oliver and J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 153-178). Sage.
Harden, A., & Thomas, J. (2005). Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methodology, 8, 257-271. https://doi.org/10.1080/13645570500155078
Hawker, S., Payne, S., Kerr, C., Hardey, M., & Powell, J. (2002). Appraising the evidence: Reviewing the disparate data systematically. Qualitative Health Research, 12, 1284-1299. https://doi.org/10.1177/1049732302238251
Heyvaert, M., Hannes, K., & Onghena, P. (2016). Using mixed methods research synthesis for literature reviews: The mixed methods research synthesis approach. Sage. https://doi.org/10.4135/9781506333243
Heyvaert, M., Maes, B., & Onghena, P. (2013). Mixed methods research synthesis: Definition, framework, and potential. Quality & Quantity, 47, 659-676. https://doi.org/10.1007/s11135-011-9538-6
Hoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., et al. (2021). Nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. Journal of Clinical Epidemiology, 138, 1–11. https://doi.org/10.1016/j.jclinepi.2021.05.022
Hong, Q.M., & Pluye, P. (2019). A conceptual framework for critical appraisal in systrematic mixed studies reviews. Journal of Mixed Methods Research, 13(4), 446-460. https://doi.org/10.1177/1558689818770058
Hong, Q.M., Pluye, P., Bujold, M., & Wassef, M. (2017). Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Systematic Reviews, 6:61. https://doi.org/10.1186/s13643-017-0454-2
JBI Manual for Evidence Synthesis (2021). CRICOS Provider Number 00123M. jbi.global
Johnson, E.E., O’Keefe, H., Sutton, A., & Marshall, C. (2022). The Systematic Review Toolbox: keeping up to date with tools to support evidence sythesis. Systematic Reviews, 11, 258. https://doi.org/10.1186/s13643-022-02122-z
Johnson, R.B., & Onwuegbuzie, A.J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33, 14-26. https://doi.org/10.3102/0013189X033007014
Johnson, R.B., Onwuegbuzie, A.J., & Turner, L.A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–33. https://doi.org/10.1177/1558689806298224
Johnstone, A., McCrorie, P., Cordovil, R., Fjørtoft, I., Livonen, S., Jidovtseff, B., Lopes, F., Reilly, J.J., Thomson, H., Wells, V., & Martin, A. (2020). Nature-based early childhood education for child health, wellbeing and development: a mixed-methods systematic review protocol. Systematic Reviews, 9, 226. https://doi.org/10.1186/s13643-020-01489-1
Jones, K. (2004). Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. The Qualitative Report, 9(1), 95-112.
Kirkham, J.J., Altman, D.G., & Williamson, P.R. (2010). Bias due to changes in specified outcomes during the systematic review process. PLoS One, 5, e9810. https://doi.org/10.1371/journal.pone.0009810
Krleza-Jeric, K., Chan, A.W., Dickersin, K., Sim, I., Grimshaw, J., & Gluud, C. (2005). Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ, 330(7497), 956–8. https://doi.org/10.1136/bmj.330.7497.956
Leslie, L., Cherry, R.F., Mulla, A., Abbott, J., Furfari, K., Glover, J.J., Hamke, B., Wynia, M.K. (2016). Domains of quality for clinical ethics case consultation: a mixed-method systematic review. Systematic Reviews, 5, 95. https://doi.org/10.1186/s13643-016-0273-x
Lockwood, C., Porrit, K., Munn, Z., Rittenmeyer, L., Salmond, S., Bjerrum, M., Loveday, H., Carrier, J, & Stannard, D. (2020). Systematic reviews of qualitative evidence. In E. Aromataris, and Z. Munn (Eds.), JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/JBIMES-20-03
Lucas, P.J., Baird, J., Arai, L., Law, C., & Roberts, H.M. (2007). Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Medical Research Methodology, 7, 4. Retrieved from http://www.biomedcentral.com/1471-2288/7/4
Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research and Policy, 10 (Suppl. 1), 6-20. https://doi.org/10.1258/1355819054308576
Maxwell, J.A. (2004a). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33, 3-11. https://doi.org/10.3102/0013189X033002003
Maxwell, J.A. (2004b). Using qualitative methods for causal explanation. Field Methods, 16, 243-264. https://doi.org/10.1177/1525822X04266831
Mazevska, D., Pearse, J, & Tierney, S. (2022). Using a theoretical framework to inform implementation of the patient-centred medical home (PCMH) model in primary care: protocol for a mixed-methods systematic review. Systematic Reviews, 11, 249. https://doi.org/10.1186/s13643-022-02132-x
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4,1. https://doi.org/10.1186/2046-4053-4-1
Morse, J.M. (1994). On the crest of a wave? (Editorial). Qualitative Health Research, 4, 139-141. https://doi.org/10.1177/104973239400400201
Munafo, M.R., Hollands, G.J., & Marteau, T.M. (2018). Open science prevents mindless science. BMJ, 363, k4309. https://doi.org/10.1136/bmj.k4309
Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Southern Illinois University Press.
Newman, I., & Hitchcock, J. H. (2011). Underlying agreements between quantitative and qualitative research: The short and tall of it all. Human Resource Development Review, 10(4), 381-398. https://doi.org/10.1177/1534484311413867
Onwuegbuzie, A. (2012). Putting the MIXED back into quantitative and qualitative research in educational reearch and beyond: Moving towards the ‘radical middle’. International Journal of Multiple Research Approaches, 6, 192-219. https://doi.org/10.5172/mra.2012.6.3.192
Onwuegbuzie, A.J. & Teddie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori & C. Teddie (Edss), Handbook of mixed metods in social and behavioral research (pp. 351-383). Sage.
Onwuegbuzie, A., & Tashakkori, A. (2015). Utilizing mixed research and evaluation methodology in peace psychology and beyond. In D. Bretherton & S. F. Law (Eds.), Methodologies in peace psychology: Peace research by peaceful means (pp. 115-137). New York, NY: Springer. https://doi.org/10.1007/978-3-319-18395-4_6
Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44(3), 795–805. https://doi.org/10.3758/s13428-011-0180-y
Pace, R., Pluye, P., Bartlett, G., Macaulay, A.C., Salsberg, J., Jagosh, J., et al. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. International Journal of Nursing Studies, 49 (1), 47–53. https://doi.org/10.1016/j.ijnurstu.2011.07.002
Page, M.J., Shamseer, L., & Tricco, A.C. (2018). Registration of systematic reviews in PROSPERO: 30,000 records and counting. Systematic Reviews, 7(1), 32. https://doi.org/10.1186/s13643-018-0699-4
Parkhurst, P.E., Lovell, K.L., Sprafka, S.A., & Hodgins, M. (1972). Evaluation of videodisc modules: A mixed methods approach. (ERIC Document Reproduction Service No. ED348014)
Pawson, R. (2006). Evidenced-based policy: A realist perspective. Sage. https://doi.org/10.4135/9781849209120
Pearson, S.A., Taylor, S, Marsden, A., & Yorke, J. (2021). Access to systemic anti-cancer therapies for women with secondary breast cancer – protocol for a mixed methods systematic review. Systematic Reviews, 10, 209. https://doi.org/10.1186/s13643-021-01761-y
PLoS Medicine Editors (2011). Best Practice in Systematic Reviews: The Importance of Protocols and Registration. PLoS Medicine, 8(2), e1001009. https://doi.org/10.1371/journal.pmed.1001009
Pluye, P. (2015). Mixed kinds of evidence: synthesis designs and critical appraisal for systematic mixed studies reviews including qualitative, quantitative and mixed methods studies. Evidence Based Medicine. Letter. https://doi.org/10.1136/ebmed-2014-110158
Pluye, P., Gagnon, M.P., Griffiths, F., Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative, and mixed methods primary-level studies in mixed studies reviews. International Journal of Nursing Studies, 46, 529-546. https://doi.org/10.1016/j.ijnurstu.2009.01.009
Pluye, P., & Hong, Q. N. (2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35, 29-45. https://doi.org/10.1146/annurev-publhealth-032013-182440
Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., et al. (2011). Proposal: A Mixed Methods Appraisal Tool for Systematic Mixed Studies Reviews. http://mixedmethodsapprai-saltoolpublic.pbworks.com
Pound, P., Britten, N., Morgan, M., Yardley, L., Pope, C., Daker-White, G., et al. (2005), Resisting medicines: A sythesis of qualitative studies of medicine taking. Social Science & Medicine, 6, 133-155. https://doi.org/10.1016/j.socscimed.2004.11.063
Preciado, M., Anguera, M.T., Olarte, M., & Lapresa, D. (2019). Observational studies in male elite football: A systematic mixed study review. Frontiers in Psychology, 10, 2077. https://doi.org/10.3389/fpsyg.2019.02077
Preciado, M., Anguera, M.T., Olarte, M., y Lapresa, D. (2021). Revisión Sistemática en Fútbol Sala desde los Mixed Methods. Revista de Psicología del Deporte / Journal of Sports Psychology, 30(1), 75-96.
Queiroga Souto, R., Khanassov, V., Hong, Q.N., Busch, P.L., Vedel, Is., & Pluye, P. (2015). Systematic mixed studies reviews: Updating results on the reliability and efficiency of the mixed methods appraisal tool. International Journal of Nursing Studies, 52, 500-501. https://doi.org/10.1016/j.ijnurstu.2014.08.010
Rana, D., Westrop, S., Germeni, E., McGarty, A., Ells, L., Lally, P., McEwan, M., Melville, C., Harris, L., & Wu, O. (2021). Understanding the effectiveness and underlying mechanisms of lifestyle modification interventions in adults with learning disabilities: protocol for a mixed-methods systematic review. Systematic Reviews, 10, 251. https://doi.org/10.1186/s13643-021-01808-0
Ridenour, C.S., & Newman, I. (2008). Mixed methods research: Exploring the interactive continuum. Southern Illinois University Press.
Sandelowski, M. (2004). Using qualitative research. Qualitative Health Research, 14, 1366-1386. https://doi.org/10.1177/1049732304269672
Sandelowski, M. (2014). Unmixing mixed-methods research. Research in Nursing & Health, 37, 3–8. https://doi.org/10.1002/nur.21570
Sandelowski, M., & Barroso, J. (2007). Handbook for synthesizing qualitative research. Springer.
Sandelowski, M, Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208-222. https://doi.org/10.1177/1558689809334210
Sandelowski, M., Voils, C.I., & Barroso, J. (2006). Defining and designing mixed research synthesis studies. Research in the Schools, 13, 29-40.
Sandelowski, M., Voils, C.J., Leeman, J., & Crandlee, J.L. (2011). Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research, 6(4), 317-331. https://doi.org/10.1177/1558689811427913.
Sarmento, H., Anguera, M.T., Pereira, A., & Araujo, D. (2018). Talent Identification and Development in Male Football: A Systematic Review. Sports Medicine, 48(4), 907-931. https://doi.org/10.1007/s40279-017-0851-7
Silagy, C.A., Middleton, P., & Hopewell, S. (2002). Publishing protocols of systematic reviews: Comparing what was done to what was planned. JAMA, 287, 2831-2834. https://doi.org/10.1001/jama.287.21.2831
Smith, V., Devane, D., Begley, C.M., & Clarke, M. (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11, 15. https://doi.org/10.1186/1471-2288-11-15
Souto, R., Khanassov, V., Pluye, P., Hong, Q.N., Bush, P., Vedel, I. (2014). Systematic mixed studies reviews: reliability testing of the Mixed Methods Appraisal Tool. In: Mixed Methods International Research Association (MMIRA) Conference. Boston, USA. http:// mixedmethodsappraisaltoolpublic.pbworks.com
Stewart, R., & Oliver, S. (2012). Making a difference with systematic reviews. In D. Gough, S. Oliver, and J. Thomas (Eds.), Introduction to Systematic Reviews (pp. 227-244). Sage.
Straus, S., & Moher, D. (2010). Registering systematic reviews. CMAJ, 182, 13-14. https://doi.org/10.1186/1471-2288-11-15
Suri, H., & Clarke, D. (2009). Advancements in research synthesis methods: From a methodologically inclusive perspective. Review of Educational Research, 79(1), 395-430. https://doi.org/10.3102/0034654308326349
Sutton, A.J., Abrams, K.R., Jones, D.R., Sheldon, T.A., & Song, F. (2000). Methods for meta-analysis in medical research. Wiley.
Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Sage.
Tashakkori, A., & Teddlie, C. (Eds.). (2010). Handbook of mixed methods in social and behavioral research (2nd ed.). Sage.
Taylor, C., & Coffey, A. (2009). Special issue: Qualitative research and methodological innovation. Qualitative Research, 9, 523-526. https://doi.org/10.1177/1468794109350355
Teddlie, C. & Tashakkori, A. (2006). A general typology of research designs featuring mixed methods. Research in the Schools, 13, 12-28.
Tennant, J.P., Waldner, F., Jacques, D.C., Masuzzo, P., Collister, L.B., Hartgerink, C.H. (2016). The academic, economic and societal impacts of open access: an evidence-based review. F1000Research, 5, 632. https://doi.org/10.12688/f1000research.8460.3.
Travers, M. (2009). New methods, old problems: A skeptical view of innovation in qualitative research. Qualitative Research, 9, 161-179. https://doi.org/10.1177/1468794108095079
Tricco, A.C., Pham, B., Brehaut, J., Tetroe, J., & Cappelli, M. (2009). An international survey indicated that unpublihed systematic reviews exist. Journal of Clinical Epidemiology, 62, 617-623. https://doi.org/10.1016/j.jclinepi.2008.09.014
Trinder, L. (2000). A critical appraisal of evidence-based practice. In L. Trinder & S. Reynols (Eds.), Evidence-based practice: A critical appraisal (pp. 212-241). Bkackwell Science. https://doi.org/10.1002/9780470699003
Trinder, L. & Reynolds, S. (Eds.) (2000). Evidence-based practice: A critical appraisal. Blackwell Science. https://doi.org/10.1002/9780470699003.ch10
Tronchoni, H., Izquierdo, C., & Anguera, M.T. (2022). A systematic review on lecturing in contemporary university teaching. Frontiers in. Psychology, 13, 971617. https://doi.org/10.3389/fpsyg.2022.971617
Tufanaru, C., Munn, Z., Aromataris, E., Campbell, J., & Hopp, L. (2020). Systematic reviews of effectiveness. In E. Aromataris, and Z. Munn (Eds.), JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/JBIMES-20-04
Van der Braak, K., Ghannad, M., Orelio, C., Heus, P., Damen, J.A.A, Spijker, R., Robinson, K., Lund, H., & Hooft, L. (2022). The score after 10 years of registration of systematic review protocols. Systematic Reviews, 11, 191. https://doi.org/10.1186/s13643-022-02053-9
Vandyk, A.D., Kaluzienski, M., Goldie, C., Stokes, Y., Ross-White, A., Kronick, J., Gilmour, M., MacPhee, C., & Graham, I.D. (2019). Interventions to improve emergency department use for mental health reasons: protocol for a mixed-methods systematic review. Systematic Reviews, 8, 84. https://doi.org/10.1186/s13643-019-1008-6
Vicente-Sáezooft, L. (2022), R., & Martínez-Fuentes, C. (2018). Open science now: a systematic literature review for an integrated definition. Journal of Business Research, 88, 428–36. https://doi.org/10.1016/j.jbusres.2017.12.043
Voils, C.I., Sandelowski, M., Barroso, J., Hasselblad, V. (2008). Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods, 20(1), 3-25. https://doi.org/10.1177/1525822X07307463
Wiafe, E., Mensah, K.B., Mensah, A.B.B., Bandalee, V., & Oosthuizen, F. (2020). The awareness of women on prostate cancer: a mixed-methods systematic review protocol. Systematic Reviews, 9, 253. https://doi.org/10.1186/s13643-020-01513-4

Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 RELIEVE – Electronic Journal of Educational Research and Evaluation

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes.
The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.
Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments.
The fact of sending your paper to RELIEVE implies that you accept these conditions.