Several studies have evaluated various ML techniques and found promising results in classifying bug reports. However, these studies have used different evaluation designs, making it difficult to compare their results. Furthermore, they have focused primarily on accuracy and did not consider other potentially relevant factors such as generalizability, explainability, and maintenance cost. These two aspects make it difficult for practitioners and researchers to choose an appropriate ML technique for a given context. Therefore, we compare promising ML techniques against practitioners’ concerns using evaluation criteria that go beyond accuracy. Based on an existing framework for adopting ML techniques, we developed an evaluation framework for ML techniques for bug report classification. We used this framework to compare nine ML techniques on three datasets. The results enable a tradeoff analysis between various promising ML techniques. The results show that an ML technique with the highest predictive accuracy might not be the most suitable technique for some contexts. The overall approach presented in the paper supports making informed decisions when choosing ML techniques. It is not locked to the specific techniques, datasets, or factors we have selected here, and others could easily use and adapt it for additional techniques or concerns. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Context: The evidence for the prevalence of test smells at the unit testing level has relied on the accuracy of detection tools, which have seen intense research in the last two decades. The Eager Test smell, one of the most prevalent, is often identified using simplified detection rules that practitioners find inadequate.

Objective: We aim to improve the rules for detecting the Eager Test smell.

Method: We reviewed the literature on test smells to analyze the definitions and detection rules of the Eager Test smell. We proposed a novel, unambiguous definition of the test smell and a heuristic to address the limitations of the existing rules. We evaluated our heuristic against existing detection rules by manually applying it to 300 unit test cases in Java.

Results: Our review identified 56 relevant studies. We found that inadequate interpretations of original definitions of the Eager Test smell led to imprecise detection rules, resulting in a high level of disagreement in detection outcomes. Also, our heuristic detected patterns of eager and non-eager tests that existing rules missed.

Conclusion: Our heuristic captures the essence of the Eager Test smell more precisely; hence, it may address practitioners’ concerns regarding the adequacy of existing detection rules.

The quality of the test suites and the constituent test cases significantly impacts confidence in software testing. While research has identified several quality attributes of test cases and test suites, there is a need for a better understanding of their relative importance in practice. We investigate practitioners' perceptions regarding the relative importance of quality attributes of test cases and test suites and the challenges that they face in ensuring the perceived important quality attributes. To capture the practitioners' perceptions, we conducted an industrial survey using a questionnaire based on the quality attributes identified in an extensive literature review. We used a sampling strategy that leverages LinkedIn to draw a large and heterogeneous sample of professionals with experience in software testing. We collected 354 responses from practitioners with a wide range of experience (from less than one year to 42 years of experience). We found that the majority of practitioners rated Fault Detection, Usability, Maintainability, Reliability, and Coverage to be the most important quality attributes. Resource Efficiency, Reusability, and Simplicity received the most divergent opinions, which, according to our analysis, depend on the software-testing contexts. Also, we identified common challenges that apply to the important attributes, namely inadequate definition, lack of useful metrics, lack of an established review process, and lack of external support. The findings point out where practitioners actually need further support with respect to achieving high-quality test cases and test suites under different software testing contexts. Hence, the findings can serve as a guideline for academic researchers when looking for research directions on the topic. Furthermore, the findings can be used to encourage companies to provide more support to practitioners to achieve high-quality test cases and test suites.

Context: Code reviewers provide valuable feedback during the code review. Identifying common issues described in the reviewers’ feedback can provide input for devising context-specific software development improvements. However, the use of reviewer feedback for this purpose is currently less explored.

Objective: In this study, we assess how automation can derive more interpretable and informative themes in reviewers’ feedback and whether these themes help to identify recurring quality-related issues in code changes.

Method: We conducted a participatory case study using the JabRef system to analyze reviewers’ feedback on merged and abandoned code changes. We used two promising topic modeling methods (GSDMM and BERTopic) to identify themes in 5,560 code review comments. The resulting themes were analyzed and named by a domain expert from JabRef.

Results: The domain expert considered the identified themes from the two topic models to represent quality-related issues. Different quality issues are pointed out in code reviews for merged and abandoned code changes. While BERTopic provides higher objective coherence, the domain expert considered themes from short-text topic modeling more informative and easy to interpret than BERTopic-based topic modeling.

Conclusions: The identified prevalent code quality issues aim to address the maintainability-focused issues. The analysis of code review comments can enhance the current practices for JabRef by improving the guidelines for new developers and focusing discussions in the developer forums. The topic model choice impacts the interpretability of the generated themes, and a higher coherence (based on objective measures) of generated topics did not lead to improved interpretability by a domain expert. 

Context: Several secondary studies have investigated the relationship between internal quality attributes, source code metrics and external quality attributes. Sometimes they have contradictory results. Objective: We synthesize evidence of the link between internal quality attributes, source code metrics and external quality attributes along with the efficacy of the prediction models used. Method: We conducted a tertiary review to identify, evaluate and synthesize secondary studies. We used several characteristics of secondary studies as indicators for the strength of evidence and considered them when synthesizing the results. Results: From 711 secondary studies, we identified 15 secondary studies that have investigated the link between source code and external quality. Our results show : (1) primarily, the focus has been on object-oriented systems, (2) maintainability and reliability are most often linked to internal quality attributes and source code metrics, with only one secondary study reporting evidence for security, (3) only a small set of complexity, coupling, and size-related source code metrics report a consistent positive link with maintainability and reliability, and (4) group method of data handling (GMDH) based prediction models have performed better than other prediction models for maintainability prediction. Conclusions: Based on our results, lines of code, coupling, complexity and the cohesion metrics from Chidamber & Kemerer (CK) metrics are good indicators of maintainability with consistent evidence from high and moderate-quality secondary studies. Similarly, four CK metrics related to coupling, complexity and cohesion are good indicators of reliability, while inheritance and certain cohesion metrics show no consistent evidence of links to maintainability and reliability. Further empirical studies are needed to explore the link between internal quality attributes, source code metrics and other external quality attributes, including functionality, portability, and usability. The results will help researchers and practitioners understand the body of knowledge on the subject and identify future research directions. © 2023 The Author(s)

Context: The adoption or acceptance of new technologies or ways of working in software development activities is a recurrent topic in the software engineering literature. The topic has, therefore, been empirically investigated extensively. It is, however, unclear which theoretical frames of reference are used in this research to explain acceptance behaviors. Objective: In this study, we explore how major theories and models of acceptance behavior have been used in the software engineering literature to empirically investigate acceptance behavior.Method: We conduct a systematic mapping study of empirical studies using acceptance behavior theories in software engineering.Results: We identified 47 primary studies covering 56 theory uses. The theories were categorized into six groups. Technology acceptance models (TAM and its extensions) were used in 29 of the 47 primary studies, innovation theories in 10, and the theories of planned behavior/ reasoned action (TPB/TRA) in six. All other theories were used in at most two of the primary studies. The usage and operationalization of the theories were, in many cases, inconsistent with the underlying theories. Furthermore, we identified 77 constructs used by these studies of which many lack clear definitions. Conclusions: Our results show that software engineering researchers are aware of some of the leading theories and models of acceptance behavior, which indicates an attempt to have more theoretical foundations. However, we identified issues related to theory usage that make it difficult to aggregate and synthesize results across studies. We propose mitigation actions that encourage the consistent use of theories and emphasize the measurement of key constructs.

Increasingly, courses on empirical software engineering research methods are being offered in higher education institutes across the world, mostly at the MSc and PhD levels. While the need for such courses is evident and in line with modern software engineering curricula, educators designing and implementing such courses have so far been reinventing the wheel; every course is designed from scratch with little to no reuse of ideas or content across the community. Due to the nature of the topic, it is rather difficult to get it right the first time when defining the learning objectives, selecting the material, compiling a reader, and, more importantly, designing relevant and appropriate practical work. This leads to substantial effort (through numerous iterations) and poses risks to the course quality. This attempts to support educators in the first and most crucial step in their course design: creating the syllabus. It does so by consolidating the collective experience of the authors as well as of members of the empirical software engineering community; the latter was mined through two working sessions and an online survey. Specifically, it offers a list of the fundamental building blocks for a syllabus, namely, course aims, course topics, and practical assignments. The course topics are also linked to the subsequent s of this book, so that readers can dig deeper into those s and get support on teaching specific research methods or cross-cutting topics. Finally, we guide educators on how to take these building blocks as a starting point and consider a number of relevant aspects to design a syllabus to meet the needs of their own program, students, and curriculum. 

Context: Evidence-based software engineering (EBSE) aims to improve research utilization in practice. It relies on systematic methods to identify, appraise, and synthesize existing research findings to answer questions of interest for practice. However, the lack of practitioners’ involvement in these studies’ design, execution, and reporting indicates a lack of appreciation for the need for knowledge exchange between researchers and practitioners. The resultant systematic literature studies often lack relevance for practice. Objective: This paper explores the use of Rapid Reviews (RRs), in fostering knowledge exchange between academia and industry. Through the lens of two case studies, we delve into the practical application and experience of conducting RRs. Methods: We analyzed the conduct of two rapid reviews by two different groups of researchers and practitioners. We collected data through interviews, and the documents produced during the review (like review protocols, search results, and presentations). The interviews were analyzed using thematic analysis. Results: We report how the two groups of researchers and practitioners performed the rapid reviews. We observed some benefits, like promoting dialogue and paving the way for future collaborations. We also found that practitioners entrusted the researchers to develop and follow a rigorous approach and were more interested in the applicability of the findings in their context. The problems investigated in these two cases were relevant but not the most immediate ones. Therefore, rapidness was not a priority for the practitioners. Conclusion: The study illustrates that rapid reviews can support researcher-practitioner communication and industry-academia collaboration. Furthermore, the recommendations based on the experiences from the two cases complement the detailed guidelines researchers and practitioners may follow to increase interaction and knowledge exchange. © 2023 The Author(s)

This handbook exploits the profound experience and expertise of well-established scholars in the empirical software engineering community to provide guidance and support in teaching various research methods and fundamental concepts. A particular focus is thus on combining research methods and their epistemological settings and terminology with didactics and pedagogy for the subject. 

The book covers the most essential contemporary research methods and philosophical and cross-cutting concerns in software engineering research, considering both academic and industrial settings, at the same time providing insights into the effective teaching of concepts and strategies.

To this end, the book is organized into four major parts. In the first part, the editors set the foundation with two chapters; one laying out the larger context of the discipline for a positioning of the remainder of this book, and one guiding the creation of a syllabus for courses in empirical software engineering. The second part of the book lays the fundamentals for teaching empirical software engineering, addressing more cross-cutting aspects from theorizing and teaching research designs to measurement and quantitative data analysis. In the third part, general experiences and personal reflections from teaching empirical software engineering in different settings are shared. Finally, the fourth part contains a number of carefully selected research methods, presented through an educational lens. Next to the chapter contributions themselves that provide a more theoretical perspective and practical advice, readers will find additional material in the form of, for example, slide sets and tools, in an online material section.

The book mainly targets three different audiences: (1) educators teaching empirical software engineering to undergraduate, postgraduate or doctoral students, (2) professional trainers teaching the basic concepts of empirical software engineering to software professionals, and (3) students and trainees attending such courses. 

Background: Modern Code Review (MCR) is a key component for delivering high-quality software and sharing knowledge among developers. Effective reviews require an in-depth understanding of the code and demand from the reviewers to contextualize the change from different perspectives.

Aim: While there is a plethora of research on solutions that support developers to understand changed code, we have observed that many provide only narrow, specialized insights and very few aggregate information in a meaningful manner. Therefore, we aim to provide a vision of improving code understanding in MCR.

Method: We classified 53 research papers suggesting proposals to improve MCR code understanding. We use this classification, the needs expressed by code reviewers from previous research, and the information we have not found in the literature for extrapolation.

Results: We identified four major types of support systems and suggest an environment for contextualized code reviews. Furthermore, we illustrate with a set of scenarios how such an environment would improve the effectiveness of code reviews.

Conclusions: Current research focuses mostly on providing narrow support for developers. We outline a vision for how MCR can be improved by using context and reducing the cognitive load on developers. We hope our vision can foster future advancements in development environments.