Context: The evidence for the prevalence of test smells at the unit testing level has relied on the accuracy of detection tools, which have seen intense research in the last two decades. The Eager Test smell, one of the most prevalent, is often identified using simplified detection rules that practitioners find inadequate.

Objective: We aim to improve the rules for detecting the Eager Test smell.

Method: We reviewed the literature on test smells to analyze the definitions and detection rules of the Eager Test smell. We proposed a novel, unambiguous definition of the test smell and a heuristic to address the limitations of the existing rules. We evaluated our heuristic against existing detection rules by manually applying it to 300 unit test cases in Java.

Results: Our review identified 56 relevant studies. We found that inadequate interpretations of original definitions of the Eager Test smell led to imprecise detection rules, resulting in a high level of disagreement in detection outcomes. Also, our heuristic detected patterns of eager and non-eager tests that existing rules missed.

Conclusion: Our heuristic captures the essence of the Eager Test smell more precisely; hence, it may address practitioners’ concerns regarding the adequacy of existing detection rules.

Context: Test cases and test suites (TCS) are central to software testing. High-quality TCS are essential for boosting practitioners’ confidence in testing. However, the quality of a test suite (a collection of test cases) is not merely the sum of the quality of individual test cases, as suite-level factors must also be considered. Achieving high-quality TCS requires defining relevant quality attributes, establishing appropriate measures for their assessment, and determining their importance within different testing contexts.

Objective: This thesis aims to (1) provide a consolidated view of TCS quality in terms of quality attributes, quality measures, and context information, (2) determine the relative importance of the quality attributes in practice, and (3) develop a reliable approach for assessing a highly prioritized quality attribute identified by practitioners.

Method: We conducted an exploratory study and a tertiary literature review for the first objective, a personal opinion survey for the second, and a comparative experiment with a small-scale evaluation study for the third.

Results: We developed a comprehensive TCS quality model grounded in practitioner insights and existing literature. Based on the survey, maintainability emerged as a critical quality attribute where practitioners need further support. A well-known indicator of poor test design that can negatively impact test-case maintainability is the Eager Test smell, which is defined as “when a test method checks several methods of the object to be tested” or “when a test verifies too much functionality.” The results of existing detection tools for eager tests are found to be inconsistent and unreliable. To better support practitioners in assessing test case maintainability, we proposed a novel, unambiguous definition of the Eager Test smell, developed a heuristic to operationalize it, and implemented a detection tool to automate its identification in practice. Our systematic approach in the tertiary review also yielded valuable insights into constructing and validating automated search results using a quasi-gold standard. We generalized these insights into recommendations for enhancing the current search validation approach.

Conclusions: The thesis makes three main contributions: (1) at the abstract level, a comprehensive quality model to help practitioners and researchers develop guidelines, templates, or tools for designing new test cases and test suites and assessing existing ones; (2) at the strategic level, identification of contextually important quality attributes; and (3), at the operational level, a refined definition of Eager Test smell, a detection heuristic and a tool prototype implementing the heuristic, advancing maintainability assessment in software testing.

The quality of the test suites and the constituent test cases significantly impacts confidence in software testing. While research has identified several quality attributes of test cases and test suites, there is a need for a better understanding of their relative importance in practice. We investigate practitioners' perceptions regarding the relative importance of quality attributes of test cases and test suites and the challenges that they face in ensuring the perceived important quality attributes. To capture the practitioners' perceptions, we conducted an industrial survey using a questionnaire based on the quality attributes identified in an extensive literature review. We used a sampling strategy that leverages LinkedIn to draw a large and heterogeneous sample of professionals with experience in software testing. We collected 354 responses from practitioners with a wide range of experience (from less than one year to 42 years of experience). We found that the majority of practitioners rated Fault Detection, Usability, Maintainability, Reliability, and Coverage to be the most important quality attributes. Resource Efficiency, Reusability, and Simplicity received the most divergent opinions, which, according to our analysis, depend on the software-testing contexts. Also, we identified common challenges that apply to the important attributes, namely inadequate definition, lack of useful metrics, lack of an established review process, and lack of external support. The findings point out where practitioners actually need further support with respect to achieving high-quality test cases and test suites under different software testing contexts. Hence, the findings can serve as a guideline for academic researchers when looking for research directions on the topic. Furthermore, the findings can be used to encourage companies to provide more support to practitioners to achieve high-quality test cases and test suites.

Context: Existing tools for detecting eager tests produce many false positives, rendering them unreliable for practitioners. To address this, our previous work introduced a novel definition of the Eager Test smell and a heuristic for more effective identification. Comparing the heuristic’s results with existing detection rules revealed eight test patterns where the rules misclassified the presence or absence of eager tests.

Objective: We aim to gather practitioners’ feedback on our heuristic’s assessment of these eight test patterns and operationalize the heuristic in a tool we named EagerID.

Method: We conducted a survey to collect practitioners’ feedback on the eight identified test patterns and developed EagerID to detect eager tests in Java unit test cases using JUnit. We also preliminarily evaluated EagerID on 300 test cases, which were manually analyzed in our previous study.

Results: Our survey received 23 responses from practitioners with a wide range of experience. We found that most practitioners agreed with the assessment of our heuristic. Furthermore, the preliminary evaluation of EagerID returned high precision (100%), recall (91.76%), and F-Score (95.70%).

Conclusion: Our survey findings highlight the practical relevance of the heuristic. The preliminary evaluation of the EagerID tool confirmed the heuristic’s potential for automation. These findings suggest that the heuristic provides a solid foundation for both manual and automated detection.

Background: Systematic literature studies (SLS) have become a core research methodology in Evidence-based Software Engineering (EBSE). Search completeness, i.e., finding all relevant papers on the topic of interest, has been recognized as one of the most commonly discussed validity issues of SLSs. Aim: This study aims at raising awareness on the issues related to search string construction and on search validation using a quasi-gold standard (QGS). Furthermore, we aim at providing guidelines for search string validation. Method: We use a recently completed tertiary study as a case and complement our findings with the observations from other researchers studying and advancing EBSE. Results: We found that the issue of assessing QGS quality has not seen much attention in the literature, and the validation of automated searches in SLSs could be improved. Hence, we propose to extend the current search validation approach by the additional analysis step of the automated search validation results and provide recommendations for the QGS construction. Conclusion: In this paper, we report on new issues which could affect search completeness in SLSs. Furthermore, the proposed guideline and recommendations could help researchers implement a more reliable search strategy in their SLSs.

Context: The core of software testing is test artifacts, i.e., test cases, test suites, test scripts, test code, test specifications, and natural language tests. Hence, the quality of test artifacts can negatively or positively impact the reliability of the software testing process. Several empirical studies and secondary studies have investigated the test artifact quality. Nevertheless, little is known about how practitioners by themselves perceive test artifact quality, and the evidence on test artifact quality in the literature has not been synthesized in one place.

Objective: This thesis aims to identify and synthesize the knowledge in test artifact quality from both academia and industry. Hence, our objectives are: (1) To understand practitioners’ perspectives on test artifact quality, (2) To investigate how test artifact quality has been characterized in the literature, (3) To increase the reliability of the research method for conducting systematic literature reviews (SLR) in software engineering.

Method: In this thesis, we conducted an interview-based exploratory study and a tertiary study to achieve the first two objectives. We used the tertiary study as a case and referred to related observations from other researchers to achieve the last objective.

Results: We provided two quality models based on the findings of the interview-based and tertiary studies. The two models were synthesized and combined to provide a broader view of test artifact quality. Also, the context information that can be used to characterize the environment in which test artifact quality is investigated was aggregated based on these studies’ findings. Based on our experience in constructing and validating automated search results using a quality gold standard (QGS) in the tertiary study, we provided recommendations for the QGS construction and proposed an extension to the current search validation approach.

Conclusions: The context information and the combined quality model provide a comprehensive view of test artifact quality. Researchers can use the quality model to develop guidelines, templates for designing new test artifacts, or assessment tools for evaluating existing test artifacts. The model also can serve as a guideline for practitioners to search for test-artifact quality information, i.e., definitions for the quality attributes and measurements. For future work, we aim at investigating how to improve relevant test artifact quality attributes that are challenging to deal with.

Context: Modern software development increasingly relies on software testing for an ever more frequent delivery of high quality software. This puts high demands on the quality of the central artifacts in software testing, test suites and test cases. Objective: We aim to develop a comprehensive model for capturing the dimensions of test case/suite quality, which are relevant for a variety of perspectives. Methods: We have carried out a systematic literature review to identify and analyze existing secondary studies on quality aspects of software testing artifacts. Results: We identified 49 relevant secondary studies. Of these 49 studies, less than half did some form of quality appraisal of the included primary studies and only 3 took into account the quality of the primary study when synthesizing the results. We present an aggregation of the context dimensions and factors that can be used to characterize the environment in which the test case/suite quality is investigated. We also provide a comprehensive model of test case/suite quality with definitions for the quality attributes and measurements based on findings in the literature and ISO/IEC 25010:2011. Conclusion: The test artifact quality model presented in the paper can be used to support test artifact quality assessment and improvement initiatives in practice. Furthermore, the model can also be used as a framework for documenting context characteristics to make research results more accessible for research and practice.

Background: Test-case quality has always been one of the major concerns in software testing. To improve test-case quality, it is important to better understand how practitioners perceive the quality of test-cases. Objective: Motivated by that need, we investigated how practitioners define test-case quality and which aspects of test-cases are important for quality assessment. Method: We conducted semi-structured interviews with professional developers, testers and test architects from a multinational software company in Sweden. Before the interviews, we asked participants for actual test cases (written in natural language) that they perceive as good, normal, and bad respectively together with rationales for their assessment. We also compared their opinions on shared test cases and contrasted their views with the relevant literature. Results: We present a quality model which consists of 11 test-case quality attributes. We also identify a misalignment in defining test-case quality among practitioners and between academia and industry, along with suggestions for improving test-case quality in industry. Conclusion: The results show that practitioners’ background, including roles and working experience, are critical dimensions of how test-case quality is defined and assessed. © Springer Nature Switzerland AG 2019.