Context: Continuous integration (CI) is a practice that aims to continuously verify quality aspects of a software intensive system both for functional and non-functional requirements (NFRs). Functional requirements are the inputs of development and can be tested in isolation, utilising either manual or automated tests. In contrast, some NFRs are difficult to test without functionality, for NFRs are often aspects of functionality and express quality aspects. Lacking this testability attribute makes NFR testing complicated and, therefore, underrepresented in industrial practice. However, the emergence of CI has radically affected software development and created new avenues for software quality evaluation and quality information acquisition. Research has, consequently, been devoted to the utilisation of this additional information for more efficient and effective NFR verification. Objective: We aim to identify the state-of-the-art of utilising the CI environment for NFR testing, hereinafter referred to as CI-NFR testing. Method: Through rigorous selection, from an initial set of 747 papers, we identified 47 papers that describe how NFRs are tested in a CI environment. Evidence-based analysis, through coding, is performed on the identified papers in this SLR. Results: Firstly, ten CI approaches are described by the papers selected, each describing different tools and nine different NFRs where reported to be tested. Secondly, although possible, CI-NFR testing is associated with eight challenges that adversely affect its adoption. Thirdly, the identified CI-NFR testing processes are tool-driven, but there is a lack of NFR testing tools that can be used in the CI environment. Finally, we proposed a CI framework for NFRs testing. Conclusion: A synthesised CI framework is proposed for testing various NFRs, and associated CI tools are also mapped. This contribution is valuable as results of the study also show that CI-NFR testing can help improve the quality of NFR testing in practices. © 2019

Context: Non-functional requirements (NFRs) (also referred to as system qualities) are essential for developing high-quality software.Notwithstanding its importance, NFR testing remains challenging, especially in terms of automation.Compared to manual verification, automated testing shows the potential to improve the efficiency and effectiveness of quality assurance, especially in the context of  Continuous Integration (CI).However, studies on how companies manage automated NFR testing through CI are limited.

Objective: This study examines how automated NFR testing can be enabledand supported using CI environments in software development companies.

Method: We performed a multi-case study at four companies by conducting 22 semi-structured interviews with industrial practitioners.

Results: Maintainability, reliability, performance, security and scalability, were found to be evaluated with automated tests in CI environments.Testing practices, quality metrics, and challenges for measuring NFRs were reported.

Conclusions: This study presents an empirically derived model that shows how data produced by CI environments can be used for evaluation and monitoring of implemented NFR quality. Additionally, the manuscript presents explicit metrics, CI components, tools, and challenges that shall be considered while performing NFR testing in practice.

The evaluation of quality attributes is often conducted in an ad-hoc manner in many companies.This practice, although often necessary to reduce cost, can lead to inefficient work and undetected quality defects.To mitigate these issues, approaches that make use of metrics/data from Continuous Integration (CI) environments have been proposed, offering many advantages, including timely feedback on code quality, early detection of quality issues, and visual representation of system quality trends.

Despite these known benefits, there is a lack of frameworks that provide guidelines how to utilize CI environments, or their components, for quality attribute evaluation.Such guidelines are important since market demands on software quality factors, e.g., security or performance, are growing.Therefore, this study aims to explore the state-of-the-practice in using CI environments and identify shared guidelines underlying industrial practices.To achieve our research objectives, we conducted a multi-case study involving four selected companies and 22 interviews with industrial participants.

In this paper, we first present a general model to offer an overview of how CI environments contribute to quality attribute evaluation.We then introduce common quality metrics used in the studied companies and from which CI components these metrics can be acquired.Finally, based on the study's findings, we propose a hierarchical and multi-layered decision support model for improving a CI environment's quality evaluation capabilities over time.