Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Method to Assess and Argue for Practical Significance in Software Engineering
Chalmers Univ Technol.ORCID iD: 0000-0002-0118-8143
USI Univ Svizzera Italiana, CHE.ORCID iD: 0000-0003-1040-3201
Chalmers Univ Technol.
Chalmers Univ Technol.ORCID iD: 0000-0001-9226-5417
Show others and affiliations
2022 (English)In: IEEE Transactions on Software Engineering, ISSN 0098-5589, E-ISSN 1939-3520, Vol. 48, no 6, p. 2053-2065Article in journal (Refereed) Published
Abstract [en]

A key goal of empirical research in software engineering is to assess practical significance, which answers the question whether the observed effects of some compared treatments show a relevant difference in practice in realistic scenarios. Even though plenty of standard techniques exist to assess statistical significance, connecting it to practical significance is not straightforward or routinely done; indeed, only a few empirical studies in software engineering assess practical significance in a principled and systematic way. In this paper, we argue that Bayesian data analysis provides suitable tools to assess practical significance rigorously. We demonstrate our claims in a case study comparing different test techniques. The case study's data was previously analyzed (Afzal et al., 2015) using standard techniques focusing on statistical significance. Here, we build a multilevel model of the same data, which we fit and validate using Bayesian techniques. Our method is to apply cumulative prospect theory on top of the statistical model to quantitatively connect our statistical analysis output to a practically meaningful context. This is then the basis both for assessing and arguing for practical significance. Our study demonstrates that Bayesian analysis provides a technically rigorous yet practical framework for empirical software engineering. A substantial side effect is that any uncertainty in the underlying data will be propagated through the statistical model, and its effects on practical significance are made clear. Thus, in combination with cumulative prospect theory, Bayesian analysis supports seamlessly assessing practical significance in an empirical software engineering context, thus potentially clarifying and extending the relevance of research for practitioners.

Place, publisher, year, edition, pages
IEEE Computer Society, 2022. Vol. 48, no 6, p. 2053-2065
Keywords [en]
Bayes methods, Data models, Software engineering, Statistical analysis, Analytical models, Testing, Decision making, Practical significance, statistical significance, Bayesian analysis, empirical software engineering
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-24154DOI: 10.1109/TSE.2020.3048991ISI: 000811580600014OAI: oai:DiVA.org:bth-24154DiVA, id: diva2:1723302
Funder
Marianne and Marcus Wallenberg Foundation, 2017.0071
Note

open access

Available from: 2023-01-03 Created: 2023-01-03 Last updated: 2023-01-03Bibliographically approved

Open Access in DiVA

fulltext(1283 kB)143 downloads
File information
File name FULLTEXT01.pdfFile size 1283 kBChecksum SHA-512
20e39a65c682113c6c9c0b15c428bee4984aa7c985ff62fbd92d7ab4af7fbf18ef5031e27a435a169a1d496c12047426fc0ef298c91903c47e0ff7cce03d6197
Type fulltextMimetype application/pdf

Other links

Publisher's full text

Authority records

Gren, Lucas

Search in DiVA

By author/editor
Torkar, RichardFuria, Carlo A.de Oliveira Neto, Francisco GomesGren, Lucas
By organisation
Department of Software Engineering
In the same journal
IEEE Transactions on Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 143 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 193 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf