Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Crossover Designs in Software Engineering Experiments: Review of the State of Analysis
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0003-3995-6125
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0002-0679-4361
Universidad Politécnica de Madrid, Spain.
2024 (English)In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, IEEE Computer Society, 2024, p. 482-488Conference paper, Published paper (Refereed)
Abstract [en]

Experimentation is an essential method for causal inference in any empirical discipline. Crossover-design experiments are common in Software Engineering (SE) research. In these, subjects apply more than one treatment in different orders. This design increases the amount of obtained data and deals with subject variability but introduces threats to internal validity like the learning and carryover effect. Vegas et al. reviewed the state of practice for crossover designs in SE research and provided guidelines on how to address its threats during data analysis while still harnessing its benefits. In this paper, we reflect on the impact of these guidelines and review the state of analysis of crossover-design experiments in SE publications between 2015 and March 2024. To this end, by conducting a forward snowballing of the guidelines, we survey 136 publications reporting 67 crossover-design experiments and evaluate their data analysis against the provided guidelines. The results show that the validity of data analyses has improved compared to the original state of analysis. Still, despite the explicit guidelines, only 29.5% of all threats to validity were addressed properly. While the maturation and the optimal sequence threats are properly addressed in 35.8% and 38.8% of all studies in our sample respectively, the carryover threat is only modeled in about 3% of the observed cases. The lack of adherence to the analysis guidelines threatens the validity of the conclusions drawn from crossover-design experiments. © 2024 Owner/Author.

Place, publisher, year, edition, pages
IEEE Computer Society, 2024. p. 482-488
Series
International Symposium on Empirical Software Engineering and Measurement, ISSN 1949-3770, E-ISSN 1949-3789
Keywords [en]
Crossover, Design, Experimentation, Literature Survey, Carry-over effects, Causal inferences, Crossover design, Design experiments, Learning effects, Software engineering experiments, Software engineering research, Design of experiments
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-27253DOI: 10.1145/3674805.3690754ISI: 001537915200048Scopus ID: 2-s2.0-85210601622ISBN: 9798400710476 (print)OAI: oai:DiVA.org:bth-27253DiVA, id: diva2:1921867
Conference
18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024, Barcelona, Oct 24-25 2024
Part of project
SERT- Software Engineering ReThought, Knowledge Foundation
Funder
Knowledge Foundation, 20180010European Regional Development Fund (ERDF)Available from: 2024-12-17 Created: 2024-12-17 Last updated: 2025-09-30Bibliographically approved
In thesis
1. Good-Enough Requirements Engineering
Open this publication in new window or tab >>Good-Enough Requirements Engineering
2025 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Background: High-quality requirements are considered crucial for successful software development endeavors as the requirements' purpose is to inform subsequent activities like implementation or testing. Requirements quality defects have been shown to incur significant costs for remediation, scaling up even to project failure. At the same time, the effort to improve the quality of requirements must be justified. Organizations developing software, therefore, need to understand when their requirements artifacts are of "good enough'' quality, i.e., they need to be able to identify the optimum between over- and under-engineering.

Problem: The body of knowledge in requirements quality does not yet offer solutions that would allow organizations to identify that optimum due to three shortcomings: (1) there is no generally accepted, theoretical foundation to describe requirements quality that can serve as a basis to coordinate distributed research efforts and the synthesis of evidence in the field, (2) the scientific practice currently applied in the field is of limited rigor to draw reliable conclusions from existing empirical contributions, and (3) the field lacks empirical evidence that can be aggregated to form a holistic view of requirements quality. These are potential causes for the lack of adoption of requirements quality research in practice.

Goal: In this cumulative, publication-based thesis, we address these three shortcomings and aim to contribute to a more evidence-based approach to requirements quality research grounded in scientific theory.

Method: First, we develop a theoretical foundation by adopting and integrating existing software engineering theories. Second, we evaluate the state of the art of data analysis and open science in the field and provide guidelines to improve these practices. Third, we demonstrate the application of these guidelines and conduct a controlled experiment to contribute additional empirical evidence to the field.

Results: The resulting set of analytical theories specifies requirements quality and provides a structure for future empirical contributions. Our evaluation of the state of the art shows both the need for a common theoretical foundation as well as support for applying rigorous research practices. Our empirical studies contribute to these needs and illustrate the complexity of the impact that requirements quality defects have on subsequent activities. Finally, we develop a method for the effective aggregation of empirical results.

Conclusion: Our theoretical, methodological, and empirical contributions help to coordinate a productive and constructive research agenda on requirements quality that is based on evidence and grounded in theory. This allows for rigorous and practically relevant research that ultimately informs organizations on how to engineer good-enough requirements.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2025. p. 257
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 2025:03
Keywords
Requirements Engineering, Requirements Artifacts, Requirements Quality
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
urn:nbn:se:bth-27382 (URN)978-91-7295-496-0 (ISBN)
Public defence
2025-02-28, J1630, Karlskrona, 13:00 (English)
Opponent
Supervisors
Available from: 2025-01-17 Created: 2025-01-16 Last updated: 2025-09-30Bibliographically approved

Open Access in DiVA

fulltext(567 kB)85 downloads
File information
File name FULLTEXT01.pdfFile size 567 kBChecksum SHA-512
e21597e62b04fd84f4a26b0438793988ea20590fdda3e57c3005f2ff7ab3bd2f57172234e8d976cd88bcf832e3e7d8b40c0ef62392dc58ef39cc28965735fabf
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Frattini, JulianFucci, Davide

Search in DiVA

By author/editor
Frattini, JulianFucci, Davide
By organisation
Department of Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 85 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 242 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf