Reporting case studies in systematic literature studies—An evidential problem
2024 (English)In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 174, article id 107501Article in journal (Refereed) Published
Abstract [en]
Context: The term and label, “case study”, is not used consistently by authors of primary studies in software engineering research. It is not clear whether this problem also occurs for systematic literature studies (SLSs).
Objective: To investigate the extent to which SLSs in/correctly use the term and label, “case study”, when classifying primary studies.
Methods: We systematically collect two sub-samples (2010–2021 & 2022) comprising a total of eleven SLSs and 79 primary studies. We examine the designs of these SLSs, and then analyse whether the SLS authors and the primary-study authors correctly label the respective primary study as a “case study”.
Results: 76% of the 79 primary studies are misclassified by SLSs (with the two sub-samples having 60% and 81% misclassification, respectively). For 39% of the 79 studies, the SLSs propagate a mislabelling by the original authors, whilst for 37%, the SLSs introduce a new mislabel, thus making the problem worse. SLSs rarely present explicit definitions for “case study” and when they do, the definition is not consistent with established definitions.
Conclusions: SLSs are both propagating and exacerbating the problem of the mislabelling of primary studies as “case studies”, rather than – as we should expect of SLSs – correcting the labelling of primary studies, and thus improving the body of credible evidence. Propagating and exacerbating mislabelling undermines the credibility of evidence in terms of its quantity, quality and relevance to both practice and research. © 2024 The Author(s)
Place, publisher, year, edition, pages
Elsevier, 2024. Vol. 174, article id 107501
Keywords [en]
Case study, Credible evidence, Systematic literature review, Systematic mapping study, Systematic review, Case-studies, Labelings, Literature studies, Misclassifications, Software engineering research, Sub-samples, Systematic mapping studies, Software engineering
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-26541DOI: 10.1016/j.infsof.2024.107501ISI: 001252262000001Scopus ID: 2-s2.0-85195473840OAI: oai:DiVA.org:bth-26541DiVA, id: diva2:1876166
2024-06-242024-06-242024-08-05Bibliographically approved