Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Automated NFR testing in Continuous Integration Environments: a multi-case study of Nordic companies
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering. Ericsson AB.ORCID iD: 0000-0001-5949-1375
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0001-7526-3727
Örebro University.
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0002-3646-235x
2023 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 28, no 6, article id 144Article in journal (Refereed) Published
Abstract [en]

Context: Non-functional requirements (NFRs) (also referred to as system qualities) are essential for developing high-quality software.Notwithstanding its importance, NFR testing remains challenging, especially in terms of automation.Compared to manual verification, automated testing shows the potential to improve the efficiency and effectiveness of quality assurance, especially in the context of  Continuous Integration (CI).However, studies on how companies manage automated NFR testing through CI are limited.

Objective: This study examines how automated NFR testing can be enabledand supported using CI environments in software development companies.

Method: We performed a multi-case study at four companies by conducting 22 semi-structured interviews with industrial practitioners.

Results: Maintainability, reliability, performance, security and scalability, were found to be evaluated with automated tests in CI environments.Testing practices, quality metrics, and challenges for measuring NFRs were reported.

Conclusions: This study presents an empirically derived model that shows how data produced by CI environments can be used for evaluation and monitoring of implemented NFR quality. Additionally, the manuscript presents explicit metrics, CI components, tools, and challenges that shall be considered while performing NFR testing in practice.

Place, publisher, year, edition, pages
Springer, 2023. Vol. 28, no 6, article id 144
Keywords [en]
Automated testing, Case study, CI, Continuous integration, Metrics, NFR, Non-functional requirements, Automation, Integration, Integration testing, Quality control, Software design, Case-studies, Continuous integrations, Integration environments, Metric, Nordic companies, System quality, Quality assurance
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-24400DOI: 10.1007/s10664-023-10356-1ISI: 001087927600001Scopus ID: 2-s2.0-85174862814OAI: oai:DiVA.org:bth-24400DiVA, id: diva2:1746896
Part of project
SERT- Software Engineering ReThought, Knowledge FoundationPLEng – Professional Licentiate of Engineering School, Knowledge Foundation
Funder
Knowledge Foundation, 20180010Knowledge Foundation, 20170213Available from: 2023-03-29 Created: 2023-03-29 Last updated: 2024-08-07Bibliographically approved
In thesis
1. Utilizing Continuous Integration environments for evaluation of software quality attributes
Open this publication in new window or tab >>Utilizing Continuous Integration environments for evaluation of software quality attributes
2023 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software quality attributes are properties that reflect the quality of a software system, and Non-functional requirements (NFRs) are the specifications that define how a software system should perform to reach a desired level of goals of the quality attributes.The evaluation of quality attributes is important to show the effectiveness of a system in meeting customers' NFRs.

Continuous integration (CI) environments have emerged as powerful platforms for organizations to improve software quality through automated software verification and validation.Despite this, there is a growing need for evaluating quality attributes that is often met by in-house development of metrics and tools.This highlights the importance of quality attributes for software product quality.

This thesis investigates the association between quality attributes and components of a CI environment, as well as how to utilize these components for evaluating software quality attributes.The focus is on improving the knowledge of the evaluation and providing specific recommendations for companies to enhance their CI environments for higher demands of quality evaluation.The contributions of this thesis include a better understanding of the relationship between quality attributes and CI components, and a set of practical guidelines for companies to effectively leverage CI for quality attribute evaluation.

The studies in this thesis utilized mixed methodologies, including a systematic literature review, a multi-case study conducted in four software development companies, and an synthesis of the collected data.The multi-case study provided a comprehensive overview of practices for quality attribute evaluation and how CI components can generate data to support the evaluation of specific attributes.The synthesis study presents a maturity model based on the collected data from both academia and industry, and the model can aid organizations in assessing their current level of maturity in utilizing CI environments to identify potential improvements.The results in these studies show the capabilities of different components of a CI environment and how these components can be used to support the evaluation of quality attributes.While the use of CI environments for the thesis topic offers benefits, it also presents several challenges, for example, the challenge to identify effective quality metrics.

In conclusion, this thesis contributes to the understanding of the use of CI environments for evaluating software quality attributes.The results suggest that CI environments can be an effective approach for quality attribute evaluation, but suitable metrics need to be considered to ensure accurate and meaningful evaluation results. Furthermore, the thesis presents areas for future research, such as the use of machine learning techniques to improve the accuracy of quality assessment using CI environments.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2023
Series
Blekinge Institute of Technology Licentiate Dissertation Series, ISSN 1650-2140 ; 5
Keywords
Quality attributes, Quality metrics, Non-functional requirement, Continuous integration, Empirical software engineering
National Category
Software Engineering
Research subject
Software Engineering
Identifiers
urn:nbn:se:bth-24402 (URN)978-91-7295-457-1 (ISBN)
Presentation
2023-06-14, J1630 + Zoom, Karlskrona, 13:00 (English)
Opponent
Supervisors
Available from: 2023-04-17 Created: 2023-04-12 Last updated: 2023-08-22Bibliographically approved

Open Access in DiVA

fulltext(1744 kB)11 downloads
File information
File name FULLTEXT02.pdfFile size 1744 kBChecksum SHA-512
77dfb860c6d2181f7c980e0a103b0bfb6f3c0f9473b2bd592d6200d630adeeabd1b80ce087ec5064cd06cb4396049cd6d1d950c652017452f45359419fcfcca6
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Yu, LiangAlégroth, EmilGorschek, Tony

Search in DiVA

By author/editor
Yu, LiangAlégroth, EmilGorschek, Tony
By organisation
Department of Software Engineering
In the same journal
Empirical Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 11 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 156 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf