Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
On the search for industry-relevant regression testing research
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0001-7266-5632
Lund University, SWE.
Halmstad University, SWE.
Halmstad University, SWE.
Show others and affiliations
2019 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 24, no 4, p. 2020-2055Article in journal (Refereed) Published
Abstract [en]

Regression testing is a means to assure that a change in the software, or

its execution environment, does not introduce new defects. It involves the expensive

undertaking of rerunning test cases. Several techniques have been proposed

to reduce the number of test cases to execute in regression testing, however, there

is no research on how to assess industrial relevance and applicability of such techniques.

We conducted a systematic literature review with the following two goals:

rstly, to enable researchers to design and present regression testing research with

a focus on industrial relevance and applicability and secondly, to facilitate the industrial

adoption of such research by addressing the attributes of concern from the

practitioners' perspective. Using a reference-based search approach, we identied

1068 papers on regression testing. We then reduced the scope to only include papers

with explicit discussions about relevance and applicability (i.e. mainly studies

involving industrial stakeholders). Uniquely in this literature review, practitioners

were consulted at several steps to increase the likelihood of achieving our aim of

identifying factors important for relevance and applicability. We have summarised

the results of these consultations and an analysis of the literature in three taxonomies,

which capture aspects of industrial-relevance regarding the regression

testing techniques. Based on these taxonomies, we mapped 38 papers reporting

the evaluation of 26 regression testing techniques in industrial settings.

Place, publisher, year, edition, pages
Springer-Verlag New York, 2019. Vol. 24, no 4, p. 2020-2055
Keywords [en]
Industrial relevance, Recommendations, Regression testing, Systematic literature review, Taxonomy
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-17364DOI: 10.1007/s10664-018-9670-1ISI: 000477582700010OAI: oai:DiVA.org:bth-17364DiVA, id: diva2:1266923
Available from: 2018-11-29 Created: 2018-11-29 Last updated: 2022-09-18Bibliographically approved
In thesis
1. Regression Testing Challenges and Solutions: An Industry-Academia Perspective
Open this publication in new window or tab >>Regression Testing Challenges and Solutions: An Industry-Academia Perspective
2019 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Background: Software quality assurance (QA) is an essential activity in the software development lifecycle. Among the different QA activities, regression testing is a challenging task for large-scale software development. Regression testing is a well-researched area, and a large number of techniques have been proposed to fulfill the needs of industry. Despite the extensive research, the adoption of proposed regression testing techniques in the industry is limited. Studies show that there is a visible gap between research and practice.

Objective: This work aims at reducing the gap between industry and academia in regression testing. To fulfill this aim we have the following objectives:

1) Understanding the practitioners' goals regarding regression testing.

2) Understanding the current state of regression testing practice and challenges in the industry.

3) Investigating the testing research applicable in an industrial context.

Method: We conducted multiple studies using different methods.

To explore the industry perspective on regression testing we used focus group and interview-based studies. To explore solutions from the literature, we used the systematic literature review and systematic mapping study.

Results: This thesis presents the practitioners' specific regression testing goals. The identified goals are confidence, controlled fault slippage, effectiveness, efficiency, and customer satisfaction. The challenges identified in the thesis are of two categories, 1) management related challenges and 2) technical challenges. Technical challenges relate to test suite maintenance, test case selection, test case prioritization, evaluation of regression testing.

We have mapped 26 empirically evaluated regression testing techniques to the context, effect, and information taxonomies, and provided a guide to the practitioners regarding the adoption of the techniques in an industrial setting. We have also classified 56 model-based test case generation techniques regarding their strengths/limitations, input/intermediate models used, and relevance to the industrial context.

Conclusions: The challenges identified in this study are not new for research and practice. There could be two reasons regarding the presence of recurring challenges: 1) regression testing techniques proposed in the literature do not fit the companies’ context, 2) or, companies are not aware of the availability of the techniques that could be suitable for their context. To support the adoption of existing research on regression testing in the industry, we have presented three taxonomies. These taxonomies, allow the characterization of regression testing techniques and enable to determine which of these techniques might be suitable in a given context. Furthermore, the identification of information needs for these techniques would be helpful to learn the implications regarding the cost of adoption. Regarding the support in test case generation, we conclude that current research on interaction model-based test case generation techniques did not illustrate the use of rigorous methodology, and currently, model-based test case generation techniques have low relevance for the industrial problems.

Place, publisher, year, edition, pages
Karlskrona, Sweden: Blekinge Tekniska Högskola, 2019. p. 146
Series
Blekinge Institute of Technology Licentiate Dissertation Series, ISSN 1650-2140 ; 2
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-17381 (URN)978-91-7295-365-9 (ISBN)
Presentation
2019-01-08, J1650, Campus Gräsvik, Karlskrona, 14:00 (English)
Opponent
Supervisors
Funder
VINNOVA, 2015-03235
Available from: 2018-12-06 Created: 2018-12-05 Last updated: 2019-01-17Bibliographically approved
2. Understanding and improving regression testing practice
Open this publication in new window or tab >>Understanding and improving regression testing practice
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Background

Regression testing is a complex and challenging activity and consumes a significant portion of software maintenance costs. Researchers are proposing various techniques to deal with the cost and complexity of regression testing. Yet, practitioners face various challenges when planning and executing regression testing. One of the main reasons is the disparity between research and practice perspectives on the goals and challenges of regression testing. In addition, it is difficult for practitioners to find techniques relevant to their context, needs, and goals because most proposed techniques lack contextual information.

Objective

This work aims to understand the challenges to regression testing practice and find ways to improve it. To fulfil this aim, we have the following objectives:

1) understanding the current state of regression testing practice, goals, and challenges,

2) finding ways to utilize regression testing research in practice, and

3) providing support in structuring and improving regression testing practice. 

Method

We have utilized various research methods, including literature reviews, workshops, focus groups, case studies, surveys, and experiments, to conduct the studies for this thesis.

Results

Research and practice stress different goals, and both follow their priorities. Researchers propose new regression testing techniques to increase the test suite's fault detection rate and maximise coverage. The practitioners consider test suite maintenance, controlled fault slippage, and confidence their priority goals. The practitioners rely on expert judgment instead of a well-defined regression testing process. They face various challenges in regression testing, such as time to test, test suit maintenance, lack of communication, lack of strategy, lack of assessment, and issues in test case selection and prioritization. 

We have proposed a GQM model representing research and practice perspectives on regression testing goals. The proposed model can help reduce disparities in research and practice perspectives and cope with the lack of assessment. 

We have created regression testing taxonomies to guide practitioners in finding techniques suitable to their product context, goals, and needs.  Further, based on the experiences of replicating a regression testing technique, we have provided guidelines for future replications and adoption of regression testing techniques.

Finally, we have designed regression testing checklists to support practitioners in decision-making while planning and performing regression testing. Practitioners who evaluated the checklists reported that the checklists covered essential aspects of regression testing and were useful and customizable to their context.

Conclusions

The thesis points out the gap in research and practice perspectives of regression testing. The regression testing challenges identified in this thesis are the evidence that either research does not consider these challenges or practitioners are unaware of how to replicate the regression testing research into their context. The GQM model presented in this thesis is a step toward reducing the research and practice gap in regression testing. Furthermore, the taxonomies and the replication experiment provide a way forward to adopting regression testing research. Finally, the checklists proposed in this thesis could help improve communication and regression test strategy. Moreover, the checklists will provide a basis for structuring and improving regression testing practice.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2022. p. 297
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 7
Keywords
Regression testing, Goals, GQM, Replication, Checklists
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
urn:nbn:se:bth-23634 (URN)978-91-7295-444-1 (ISBN)
Public defence
2022-10-31, C413A, Campus Grasvik, Karlskrona, 13:00 (English)
Opponent
Supervisors
Available from: 2022-09-20 Created: 2022-09-18 Last updated: 2022-10-10Bibliographically approved

Open Access in DiVA

fulltext(2309 kB)320 downloads
File information
File name FULLTEXT02.pdfFile size 2309 kBChecksum SHA-512
0ac506cc1c60b33022413eb9436ea699d488498188bf2f61ad39b0efc900645a2472e449715e1897de541a3c069f05187c586484e19468a33a60c8c1d807d1cf
Type fulltextMimetype application/pdf

Other links

Publisher's full text

Authority records

Ali, Nauman bin

Search in DiVA

By author/editor
Ali, Nauman binMinhas, Nasir Mehmood
By organisation
Department of Software Engineering
In the same journal
Empirical Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 478 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 998 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf