Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Understanding and improving regression testing practice
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0001-8177-4355
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Background

Regression testing is a complex and challenging activity and consumes a significant portion of software maintenance costs. Researchers are proposing various techniques to deal with the cost and complexity of regression testing. Yet, practitioners face various challenges when planning and executing regression testing. One of the main reasons is the disparity between research and practice perspectives on the goals and challenges of regression testing. In addition, it is difficult for practitioners to find techniques relevant to their context, needs, and goals because most proposed techniques lack contextual information.

Objective

This work aims to understand the challenges to regression testing practice and find ways to improve it. To fulfil this aim, we have the following objectives:

1) understanding the current state of regression testing practice, goals, and challenges,

2) finding ways to utilize regression testing research in practice, and

3) providing support in structuring and improving regression testing practice. 

Method

We have utilized various research methods, including literature reviews, workshops, focus groups, case studies, surveys, and experiments, to conduct the studies for this thesis.

Results

Research and practice stress different goals, and both follow their priorities. Researchers propose new regression testing techniques to increase the test suite's fault detection rate and maximise coverage. The practitioners consider test suite maintenance, controlled fault slippage, and confidence their priority goals. The practitioners rely on expert judgment instead of a well-defined regression testing process. They face various challenges in regression testing, such as time to test, test suit maintenance, lack of communication, lack of strategy, lack of assessment, and issues in test case selection and prioritization. 

We have proposed a GQM model representing research and practice perspectives on regression testing goals. The proposed model can help reduce disparities in research and practice perspectives and cope with the lack of assessment. 

We have created regression testing taxonomies to guide practitioners in finding techniques suitable to their product context, goals, and needs.  Further, based on the experiences of replicating a regression testing technique, we have provided guidelines for future replications and adoption of regression testing techniques.

Finally, we have designed regression testing checklists to support practitioners in decision-making while planning and performing regression testing. Practitioners who evaluated the checklists reported that the checklists covered essential aspects of regression testing and were useful and customizable to their context.

Conclusions

The thesis points out the gap in research and practice perspectives of regression testing. The regression testing challenges identified in this thesis are the evidence that either research does not consider these challenges or practitioners are unaware of how to replicate the regression testing research into their context. The GQM model presented in this thesis is a step toward reducing the research and practice gap in regression testing. Furthermore, the taxonomies and the replication experiment provide a way forward to adopting regression testing research. Finally, the checklists proposed in this thesis could help improve communication and regression test strategy. Moreover, the checklists will provide a basis for structuring and improving regression testing practice.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2022. , p. 297
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 7
Keywords [en]
Regression testing, Goals, GQM, Replication, Checklists
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
URN: urn:nbn:se:bth-23634ISBN: 978-91-7295-444-1 (print)OAI: oai:DiVA.org:bth-23634DiVA, id: diva2:1696627
Public defence
2022-10-31, C413A, Campus Grasvik, Karlskrona, 13:00 (English)
Opponent
Supervisors
Available from: 2022-09-20 Created: 2022-09-18 Last updated: 2022-10-10Bibliographically approved
List of papers
1. Regression testing goals: View of practitioners and researchers
Open this publication in new window or tab >>Regression testing goals: View of practitioners and researchers
2017 (English)In: 24th Asia-Pacific Software Engineering Conference Workshops (APSECW), IEEE, 2017, p. 25-32Conference paper, Published paper (Refereed)
Abstract [en]

Context: Regression testing is a well-researched area. However, the majority regression testing techniques proposed by the researchers are not getting the attention of the practitioners. Communication gaps between industry and academia and disparity in the regression testing goals are the main reasons. Close collaboration can help in bridging the communication gaps and resolving the disparities.Objective: The study aims at exploring the views of academics and practitioners about the goals of regression testing. The purpose is to investigate the commonalities and differences in their viewpoints and defining some common goals for the success of regression testing.Method: We conducted a focus group study, with 7 testing experts from industry and academia. 4 testing practitioners from 2companies and 3 researchers from 2 universities participated in the study. We followed GQM approach, to elicit the regression testing goals, information needs, and measures.Results: 43 regression testing goals were identified by the participants, which were reduced to 10 on the basis of similarity among the identified goals. Later during the priority assignment process, 5 goals were discarded, because the priority assigned to these goals was very low. Participants identified 47 information needs/questions required to evaluate the success of regression testing with reference to goal G5 (confidence). Which were then reduced to10 on the basis of similarity. Finally, we identified measures to gauge those information needs/questions, which were corresponding to the goal (G5).Conclusions: We observed that participation level of practitioners and researchers during the elicitation of goals and questions was same. We found a certain level of agreement between the participants regarding the regression testing definitions and goals.But there was some level of disagreement regarding the priorities of the goals. We also identified the need to implement a regression testing evaluation framework in the participating companies.

Place, publisher, year, edition, pages
IEEE, 2017
Keywords
Regression testing, Regression testing goals, GQM, Focus group
National Category
Computer Systems
Identifiers
urn:nbn:se:bth-16009 (URN)10.1109/APSECW.2017.23 (DOI)000428319200008 ()978-1-5386-2649-8 (ISBN)
Conference
24th Asia-Pacific Software Engineering Conference, Nanjing
Projects
EASE (Embedded Applications Software Engineering, ease.cs.lth.se)
Available from: 2018-03-22 Created: 2018-03-22 Last updated: 2022-09-18Bibliographically approved
2. Using goal-question-metric to Compare Research and Practice Perspectives on Regression Testing
Open this publication in new window or tab >>Using goal-question-metric to Compare Research and Practice Perspectives on Regression Testing
2023 (English)In: Journal of Software: Evolution and Process, ISSN 2047-7473, E-ISSN 2047-7481, Vol. 35, no 2, article id e2506Article in journal (Refereed) Published
Abstract [en]

Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large-scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. 

This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. 

We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. 

To bridge the research and practice gap, based on the literature and survey findings, we have created a goal-question-metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals. 

Place, publisher, year, edition, pages
John Wiley & Sons, 2023
Keywords
Regression testing, Goals, Objectives, Measures, Metrics, GQM
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
urn:nbn:se:bth-23630 (URN)10.1002/smr.2506 (DOI)000852963100001 ()2-s2.0-85137875656 (Scopus ID)
Funder
ELLIIT - The Linköping‐Lund Initiative on IT and Mobile Communications
Available from: 2022-09-13 Created: 2022-09-13 Last updated: 2023-06-19Bibliographically approved
3. Regression testing for large-scale embedded software development: Exploring the state of practice
Open this publication in new window or tab >>Regression testing for large-scale embedded software development: Exploring the state of practice
2020 (English)In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 120, article id UNSP 106254Article in journal (Refereed) Published
Abstract [en]

Context: A majority of the regression testing techniques proposed by the research have not been adopted in industry. To increase adoption rates, we need to better understand the practitioners' perspectives on regression testing.

Objective: This study aims at exploring the regression testing state of practice in the large-scale embedded software development. The study has two objectives, 1) to highlight the potential challenges in practice, and 2) to identify the industry-relevant research areas regarding regression testing.

Method: We conducted a qualitative study in two large-scale embedded software development companies, where we carried out semi-structured interviews with representatives from five software testing teams. We did conduct the detailed review of the process documentation of the companies to complement/validate the findings of the interviews.

Results: Mostly, the practitioners run regression testing with a selected scope, the selection of scope depends upon the size, complexity, and location of the change. Test cases are prioritized on the basis of risk and critical functionality. The practitioners rely on their knowledge and experience for the decision making regarding selection and prioritization of test cases.The companies are using both automated and manual regression testing, and mainly they rely on in-house developed tools for test automation. The challenges identified in the companies are: time to test, information management, test suite maintenance, lack of communication, test selection/prioritization, lack of assessment, etc. The proposed improvements are in line with the identified challenges. Regression testing goals identified in this study are customer satisfaction, critical defect detection, confidence, effectiveness, efficiency, and controlled slip through of faults.

Conclusions: Considering the current state of practice and identified challenges we conclude that there is a need to reconsider the regression test strategy in the companies. Researchers need to analyze the industry perspective while proposing new regression testing techniques. The industry-academia collaboration projects would be a good platform in this regard.

Place, publisher, year, edition, pages
Elsevier, 2020
Keywords
Regression testing, practices, challenges, goals, multi-case study
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-17362 (URN)10.1016/j.infsof.2019.106254 (DOI)000514233800006 ()
Projects
EASE - Embedded Applications Software Engineering
Funder
Vinnova, 2015-03235
Available from: 2018-11-30 Created: 2018-11-30 Last updated: 2022-09-18Bibliographically approved
4. On the search for industry-relevant regression testing research
Open this publication in new window or tab >>On the search for industry-relevant regression testing research
Show others...
2019 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 24, no 4, p. 2020-2055Article in journal (Refereed) Published
Abstract [en]

Regression testing is a means to assure that a change in the software, or

its execution environment, does not introduce new defects. It involves the expensive

undertaking of rerunning test cases. Several techniques have been proposed

to reduce the number of test cases to execute in regression testing, however, there

is no research on how to assess industrial relevance and applicability of such techniques.

We conducted a systematic literature review with the following two goals:

rstly, to enable researchers to design and present regression testing research with

a focus on industrial relevance and applicability and secondly, to facilitate the industrial

adoption of such research by addressing the attributes of concern from the

practitioners' perspective. Using a reference-based search approach, we identied

1068 papers on regression testing. We then reduced the scope to only include papers

with explicit discussions about relevance and applicability (i.e. mainly studies

involving industrial stakeholders). Uniquely in this literature review, practitioners

were consulted at several steps to increase the likelihood of achieving our aim of

identifying factors important for relevance and applicability. We have summarised

the results of these consultations and an analysis of the literature in three taxonomies,

which capture aspects of industrial-relevance regarding the regression

testing techniques. Based on these taxonomies, we mapped 38 papers reporting

the evaluation of 26 regression testing techniques in industrial settings.

Place, publisher, year, edition, pages
Springer-Verlag New York, 2019
Keywords
Industrial relevance, Recommendations, Regression testing, Systematic literature review, Taxonomy
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-17364 (URN)10.1007/s10664-018-9670-1 (DOI)000477582700010 ()
Available from: 2018-11-29 Created: 2018-11-29 Last updated: 2022-09-18Bibliographically approved
5. Lessons learned from replicating a study on information-retrieval based test case prioritization
Open this publication in new window or tab >>Lessons learned from replicating a study on information-retrieval based test case prioritization
2023 (English)In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367, Vol. 31, no 4, p. 1527-1559Article in journal (Refereed) Published
Abstract [en]

Replication studies help solidify and extend knowledge by evaluating previous studies’ findings. Software engineering literature showed that too few replications are conducted focusing on software artifacts without the involvement of humans. This study aims to replicate an artifact-based study on software testing to address the gap related to replications. In this investigation, we focus on (i) providing a step-by-step guide of the replication, reflecting on challenges when replicating artifact-based testing research and (ii) evaluating the replicated study concerning the validity and robustness of the findings. We replicate a test case prioritization technique proposed by Kwon et al. We replicated the original study using six software programs, four from the original study and two additional software programs. We automated the steps of the original study using a Jupyter notebook to support future replications. Various general factors facilitating replications are identified, such as (1) the importance of documentation; (2) the need for assistance from the original authors; (3) issues in the maintenance of open-source repositories (e.g., concerning needed software dependencies, versioning); and (4) availability of scripts. We also noted observations specific to the study and its context, such as insights from using different mutation tools and strategies for mutant generation. We conclude that the study by Kwon et al. is partially replicable for small software programs and could be automated to facilitate software practitioners, given the availability of required information. However, it is hard to implement the technique for large software programs with the current guidelines. Based on lessons learned, we suggest that the authors of original studies need to publish their data and experimental setup to support the external replications. © 2023, The Author(s).

Place, publisher, year, edition, pages
Springer, 2023
Keywords
Replication, Regression testing, Technique, Test case prioritization, Information retrieval, SIR
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-23631 (URN)10.1007/s11219-023-09650-4 (DOI)001084224100001 ()2-s2.0-85174265778 (Scopus ID)
Funder
ELLIIT - The Linköping‐Lund Initiative on IT and Mobile Communications
Available from: 2022-09-13 Created: 2022-09-13 Last updated: 2023-12-05Bibliographically approved
6. Checklists to support decision-making in regression testing
Open this publication in new window or tab >>Checklists to support decision-making in regression testing
2023 (English)In: Journal of Systems and Software, ISSN 0164-1212, E-ISSN 1873-1228, Vol. 202, article id 111697Article in journal (Refereed) Published
Abstract [en]

Context: Practitioners working in large-scale software development face many challenges in regression testing activities. One of the reasons is the lack of a structured regression testing process. In this regard, checklists can help practitioners keep track of essential regression testing activities and add structure to the regression testing process to a certain extent. Objective: This study aims to introduce regression testing checklists so test managers/teams can use them: (1) to assess whether test teams/members are ready to begin regression testing, and (2) to keep track of essential regression testing activities while planning and executing regression tests. Method: We used interviews, workshops, and questionnaires to design, evolve, and evaluate regression testing checklists. In total, 25 practitioners from 12 companies participated in creating the checklist. Twenty-three of them participated in checklists evolution and evaluation. Results: We identified activities practitioners consider significant while planning, performing, and analyzing regression testing. We designed regression testing checklists based on these activities to help practitioners make informed decisions during regression testing. With the help of practitioners, we evolved these checklists into two iterations. Finally, the practitioners provided feedback on the proposed checklists. All respondents think the proposed checklists are useful and customizable for their environments, and 80% think checklists cover aspects essential for regression testing. Conclusion: The proposed regression testing checklists can be useful for test managers to assess their team/team members’ readiness and decide when to start and stop regression testing. The checklists can be used to record the steps required while planning and executing regression testing. Further, these checklists can provide a basis for structuring the regression testing process in varying contexts. © 2023 The Author(s)

Place, publisher, year, edition, pages
Elsevier, 2023
Keywords
Regression testing, Checklists, Test manager, Team readiness, Process improvement
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
urn:nbn:se:bth-23675 (URN)10.1016/j.jss.2023.111697 (DOI)000989289800001 ()2-s2.0-85153245617 (Scopus ID)
Funder
ELLIIT - The Linköping‐Lund Initiative on IT and Mobile Communications
Available from: 2022-09-18 Created: 2022-09-18 Last updated: 2023-06-12Bibliographically approved

Open Access in DiVA

fulltext(4907 kB)878 downloads
File information
File name FULLTEXT03.pdfFile size 4907 kBChecksum SHA-512
a3595f75dc881b8ee92e7b697e842dc4a25fda8729b14968efdc695e09b48087ceaf491d201dede7467b827469b7943bf0b51a32bd53b95eaf549fc2b3959dce
Type fulltextMimetype application/pdf

Authority records

Minhas, Nasir Mehmood

Search in DiVA

By author/editor
Minhas, Nasir Mehmood
By organisation
Department of Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 923 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1159 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf