Quality attributes of software systems, also known as system qualities, such as performance, security, and scalability, continue to grow in importance in industrial practice. The evaluation of quality attributes is critical to software development since optimizing a software system’s core attributes can provide marketing advantage and set a product apart from its competitors. Many existing studies of unsuccessful development projects report that lack of quality attribute evaluation is often a contributing factor of project failure. Therefore, continuous quality attribute evaluation, throughout the development process, is needed to ensure customers’ expectations and demands are met.
Manual evaluation of software attributes is common in many software development companies, but it has proven to be insufficient in meeting the demands of rapid releases and high-quality expectations from customers. Automated practices have therefore gained widespread popularity as a solution to enhance efficiency, reduce costs, and increase accuracy compared to manual evaluation.
One way to automate the evaluation is using continuous integration (CI) environments. The CI environment provides several benefits, such as fast feedback on code quality, early detection of quality defects, and visualization of system quality trends. As such, these environments inherently offer organizations the opportunity to continuously monitor the quality of their software systems. However, an immature automation process can result in negative outcomes, such as cost and schedule overruns, slow feedback loops, and delayed releases.
To improve the evaluation process, prior studies have investigated different key areas, including knowledge, processes, tools, and metrics. While leveraging these areas can have a positive impact on quality evaluation, to the best of our knowledge, there is a lack of frameworks that link CI environment knowledge, metrics, and evolution together.
In this article, we aim to fill this gap by presenting the state-of-practice of using CI environments for the evaluation of quality attributes. This is achieved through an industrial study at four partner companies. Study results show that metrics acquired from CI components have a positive effect on evaluating quality requirements. Through analyzing these results, we propose a model by providing guidelines to mature existing CI environments that organizations can use for quality improvements.
As such, we claim the following contributions of this study:
A generic model of how CI environments contribute to quality attribute evaluation.
Empirical evidence that demonstrates how CI components can be used to produce data supporting the evaluation of quality attributes with metrics.
A model, derived from the study results, which provides decision support to evolve software quality evaluation through CI environments over time. © 2024 Owner/Author.