Context: Anomaly detection is crucial for maintaining cloud-based software systems, as it enables early identification and resolution of unexpected failures. Given rapid and significant advances in the anomaly detection domain and the complexity of its industrial implementation, an overview of techniques that utilize real-world operational data is needed.

Aim: This study aims to complement existing research with an extensive catalog of the techniques and monitoring data used for detecting anomalies affecting the performance or reliability of cloud-based software systems that have been developed and/or evaluated in a real-world context.

Method: We perform a systematic mapping study to examine the literature on anomaly detection in cloud-based systems, particularly focusing on the usage of real-world monitoring data, with the aim of identifying key data categories, tools, data preprocessing, and anomaly detection techniques.

Results: Based on a review of 104 papers, we categorize monitoring data by structure, types, and origins and the tools used for data collection and processing. We offer a comprehensive overview of data preprocessing and anomaly detection techniques mapped to different data categories. Our findings highlight practical challenges and considerations in applying these techniques in real-world cloud environments.

Conclusion: The findings help practitioners and researchers identify relevant data categories and select appropriate data preprocessing and anomaly detection techniques for their specific operational environments, which is important for improving the reliability and performance of cloud-based systems.

Software startups operate in uncertain environments that require agile methodologies that support adaptability. Although Scrum and Kanban are widely adopted in software startups, their contributions to managing uncertainty and adaptability remain underexplored. This study investigates which aspects of Scrum and Kanban are the most effective in addressing uncertainty and volatility to improve adaptability in software startups. The goal is to identify key practices that improve agility and examine how startups tailor agile methodologies, particularly where they face limitations. To achieve this, a mixed methods approach was used, combining a literature review with a survey. The survey collected responses from 121 startup professionals to assess their experiences with Scrum and Kanban in handling uncertainty, workload management, workflow visualization, iteration planning, risk management, and testing. The results show that iterative development and a sustainable pace in it, prioritization, and visualization improve agility and productivity. Scrum supports structured iteration and events, while Kanban enhances workflow transparency and flow management. However, limitations in risk management, team structure, collaboration between business and technical people, and testing suggest agile frameworks require further adaptation in startups. In general, the study reinforces the need for context-specific adaptations of agile methods, as no single approach fully addresses the adaptability and uncertainty of startup environments. It highlights pivotal aspects of Scrum and Kanban to construct hybrid or customized approaches for complex, fast-paced software environments. 

Software analytics (SA) is often proposed as a tool to support software engineering (SE) tasks. Several secondary studies on SA have been published, some published within the same calendar year. This presents an opportunity to take a meta-perspective and examine how the field of SA has been conceptualized and synthesized so far. By analyzing how SA is defined, which topics are emphasized, what search strategies are employed, and to what extent primary studies overlap, we aim to identify gaps, trends, and redundancies in the current body of secondary studies. Such insights can inform the design and focus of future secondary studies. We identified five secondary studies on SA published from 2015 to 2023 that cover primary research from 2000 to 2021. Despite similarities in objectives and overlapping search timeframes, the secondary studies have negligible overlap in their included primary studies. Each secondary study presents a distinct perspective, and collectively, the five secondary studies offer a fragmented rather than cohesive view of the research landscape. We present a structured overview of the identified secondary studies in terms of their objectives, research quality, and findings. This overview helps readers navigate and leverage existing research. The analysis also indicates that there is potential for further secondary research to build a more cohesive and comprehensive understanding of the SA literature. 

Several studies have evaluated various ML techniques and found promising results in classifying bug reports. However, these studies have used different evaluation designs, making it difficult to compare their results. Furthermore, they have focused primarily on accuracy and did not consider other potentially relevant factors such as generalizability, explainability, and maintenance cost. These two aspects make it difficult for practitioners and researchers to choose an appropriate ML technique for a given context. Therefore, we compare promising ML techniques against practitioners’ concerns using evaluation criteria that go beyond accuracy. Based on an existing framework for adopting ML techniques, we developed an evaluation framework for ML techniques for bug report classification. We used this framework to compare nine ML techniques on three datasets. The results enable a tradeoff analysis between various promising ML techniques. The results show that an ML technique with the highest predictive accuracy might not be the most suitable technique for some contexts. The overall approach presented in the paper supports making informed decisions when choosing ML techniques. It is not locked to the specific techniques, datasets, or factors we have selected here, and others could easily use and adapt it for additional techniques or concerns. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Context: The evidence for the prevalence of test smells at the unit testing level has relied on the accuracy of detection tools, which have seen intense research in the last two decades. The Eager Test smell, one of the most prevalent, is often identified using simplified detection rules that practitioners find inadequate.

Objective: We aim to improve the rules for detecting the Eager Test smell.

Method: We reviewed the literature on test smells to analyze the definitions and detection rules of the Eager Test smell. We proposed a novel, unambiguous definition of the test smell and a heuristic to address the limitations of the existing rules. We evaluated our heuristic against existing detection rules by manually applying it to 300 unit test cases in Java.

Results: Our review identified 56 relevant studies. We found that inadequate interpretations of original definitions of the Eager Test smell led to imprecise detection rules, resulting in a high level of disagreement in detection outcomes. Also, our heuristic detected patterns of eager and non-eager tests that existing rules missed.

Conclusion: Our heuristic captures the essence of the Eager Test smell more precisely; hence, it may address practitioners’ concerns regarding the adequacy of existing detection rules.

Background: Systematic literature reviews (SLRs) have become prevalent in software engineering research. Several researchers may conduct SLRs on similar topics without a prospective register for SLR protocols. However, even ignoring these unavoidable duplications of effort in the simultaneous conduct of SLRs, the proliferation of overlapping and often repetitive SLRs indicates that researchers are not extensively checking for existing SLRs on a topic. Given how effort-intensive it is to design, conduct, and report an SLR, the situation is less than ideal for software engineering research.

Aim: To understand how authors justify additional SLRs on a topic.

Method: To illustrate the issue and develop suggestions for improvement to address this issue, we have intentionally picked a sufficiently narrow but well-researched topic, i.e., effort estimation in Agile software development. We identify common justification patterns through a qualitative content analysis of 18 published SLRs. We further consider the citation data, publication years, publication venues, and the quality of the SLRs when interpreting the results.

Results: The common justification patterns include authors claiming gaps in coverage, methodological limitations in prior studies, temporal obsolescence of previous SLRs, or rapid technological and methodological advancements necessitating updated syntheses.

Conclusion: Our in-depth analysis of SLRs on a fairly narrow topic provides insights into SLRs in software engineering in general. By emphasizing the need for identifying existing SLRs and for justifying the undertaking of further SLRs, both in design and review guidelines and as a policy of conferences and journals, we can reduce the likelihood of duplication of effort and increase the rate of progress in the field. 

Context: Systematic literature studies, such as secondary studies, are crucial to aggregate evidence. An essential part of these studies is the selection phase of relevant studies. This, however, is time-consuming, resource-intensive, and error-prone as it highly depends on manual labor and domain expertise. The increasing popularity of Large Language Models (LLMs) raises the question to what extent these manual study selection tasks could be supported in an automated manner.

Objectives: In this manuscript, we report on our effort to explore and evaluate the use of state-of-the-art LLMs to automate the selection phase in systematic literature studies.

Method: We evaluated LLMs for the selection phase using two published systematic literature studies in software engineering as ground truth. Three prompts were designed and applied across five LLMs to the studies’ titles and abstracts based on their inclusion and exclusion criteria. Additionally, we analyzed combining two LLMs to replicate a practical selection phase. We analyzed recall and precision and reflected upon the accuracy of the LLMs, and whether the ground truth studies were conducted by early career scholars or by more advanced ones.

Results: Our results show a high average recall of up to 98% combined with a precision of 27% in a single LLM approach and an average recall of 99% with a precision of 27% in a two-model approach replicating a two-reviewer procedure. Further the Llama 2 models showed the highest average recall 98% across all prompt templates and datasets while GPT4-turbo had the highest average precision 72%.

Conclusions: Our results demonstrate how LLMs could support a selection phase in the future. We recommend a two LLM-approach to archive a higher recall. However, we also critically reflect upon how further studies are required using other models and prompts on more datasets to strengthen the confidence in our presented approach. © 2025 The Authors

Systematic Mapping Studies (SMSs) are valuable in evidence-based software engineering research. SMSs aim to provide an overview of research, identify gaps and trends, and assess the feasibility of conducting a more focused systematic literature review. In current guidelines for conducting SMSs, a quality assessment of the included papers is suggested only when the research questions explicitly require such a quality assessment. We agree with the recommendation that quality assessment is generally non-mandatory. However, SMSs deal with papers ranging from opinion papers to papers reporting highly rigorous empirical studies. Therefore, in this paper, we argue that analyzing the type of papers is essential for almost every intended purpose of an SMS. Otherwise, without distinguishing papers based on their types, we risk deriving a less informative or incomplete overview or, at worst, a misleading overview of research. Petersen et al. 'encourage' the classification of papers into six paper types as proposed by Wieringa et al.: evaluation research, solution proposal, validation research, philosophical papers, opinion papers, and personal experience papers. Given the lenient guidelines on assessing the quality of included studies, we recommend a stronger focus on classifying papers by type. 

The quality of the test suites and the constituent test cases significantly impacts confidence in software testing. While research has identified several quality attributes of test cases and test suites, there is a need for a better understanding of their relative importance in practice. We investigate practitioners' perceptions regarding the relative importance of quality attributes of test cases and test suites and the challenges that they face in ensuring the perceived important quality attributes. To capture the practitioners' perceptions, we conducted an industrial survey using a questionnaire based on the quality attributes identified in an extensive literature review. We used a sampling strategy that leverages LinkedIn to draw a large and heterogeneous sample of professionals with experience in software testing. We collected 354 responses from practitioners with a wide range of experience (from less than one year to 42 years of experience). We found that the majority of practitioners rated Fault Detection, Usability, Maintainability, Reliability, and Coverage to be the most important quality attributes. Resource Efficiency, Reusability, and Simplicity received the most divergent opinions, which, according to our analysis, depend on the software-testing contexts. Also, we identified common challenges that apply to the important attributes, namely inadequate definition, lack of useful metrics, lack of an established review process, and lack of external support. The findings point out where practitioners actually need further support with respect to achieving high-quality test cases and test suites under different software testing contexts. Hence, the findings can serve as a guideline for academic researchers when looking for research directions on the topic. Furthermore, the findings can be used to encourage companies to provide more support to practitioners to achieve high-quality test cases and test suites.

Context: Code reviewers provide valuable feedback during the code review. Identifying common issues described in the reviewers’ feedback can provide input for devising context-specific software development improvements. However, the use of reviewer feedback for this purpose is currently less explored.

Objective: In this study, we assess how automation can derive more interpretable and informative themes in reviewers’ feedback and whether these themes help to identify recurring quality-related issues in code changes.

Method: We conducted a participatory case study using the JabRef system to analyze reviewers’ feedback on merged and abandoned code changes. We used two promising topic modeling methods (GSDMM and BERTopic) to identify themes in 5,560 code review comments. The resulting themes were analyzed and named by a domain expert from JabRef.

Results: The domain expert considered the identified themes from the two topic models to represent quality-related issues. Different quality issues are pointed out in code reviews for merged and abandoned code changes. While BERTopic provides higher objective coherence, the domain expert considered themes from short-text topic modeling more informative and easy to interpret than BERTopic-based topic modeling.

Conclusions: The identified prevalent code quality issues aim to address the maintainability-focused issues. The analysis of code review comments can enhance the current practices for JabRef by improving the guidelines for new developers and focusing discussions in the developer forums. The topic model choice impacts the interpretability of the generated themes, and a higher coherence (based on objective measures) of generated topics did not lead to improved interpretability by a domain expert.