Large Language Models (LLMs) offer promising capabilities for information retrieval and processing. However, the LLM deployment for querying proprietary enterprise data poses unique challenges, particularly for companies with strict data security policies. This study shares our experience in setting up a secure LLM environment within a FinTech company and utilizing it for enterprise information retrieval while adhering to data privacy protocols. 

We conducted three workshops and 30 interviews with industrial engineers to gather data and requirements. The interviews further enriched the insights collected from the workshops. We report the steps to deploy an LLM solution in an industrial sandboxed environment and lessons learned from the experience. These lessons contain LLM configuration (e.g., chunk_size and top_k settings), local document ingestion, and evaluating LLM outputs.

Our lessons learned serve as a practical guide for practitioners seeking to use private data with LLMs to achieve better usability, improve user experiences, or explore new business opportunities. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Context: As the diversity and complexity of regulations affecting Software-Intensive Products and Services (SIPS) is increasing, software engineers need to address the growing regulatory scrutiny. We argue that, as with any other non-negotiable requirements, SIPS compliance should be addressed early in SIPS engineering—i.e., during requirements engineering (RE).

Objectives: In the conditions of the expanding regulatory landscape, existing research offers scattered insights into regulatory compliance of SIPS. This study addresses the pressing need for a structured overview of the state of the art in software RE and its contribution to regulatory compliance of SIPS.

Method: We conducted a systematic mapping study to provide an overview of the current state of research regarding challenges, principles, and practices for regulatory compliance of SIPS related to RE. We focused on the role of RE and its contribution to other SIPS lifecycle process areas. We retrieved 6914 studies published from 2017 (January 1) until 2023 (December 31) from four academic databases, which we filtered down to 280 relevant primary studies.

Results: We identified and categorized the RE-related challenges in regulatory compliance of SIPS and their potential connection to six types of principles and practices addressing challenges. We found that about 13.6% of the primary studies considered the involvement of both software engineers and legal experts in developing principles and practices. About 20.7% of primary studies considered RE in connection to other process areas. Most primary studies focused on a few popular regulation fields (privacy, quality) and application domains (healthcare, software development, avionics). Our results suggest that there can be differences in terms of challenges and involvement of stakeholders across different fields of regulation.

Conclusion: Our findings highlight the need for an in-depth investigation of stakeholders’ roles, relationships between process areas, and specific challenges for distinct regulatory fields to guide research and practice. 

Organisations are becoming concerned with effectively dealing with privacy-related requirements. Existing Requirements Engineering methods based on structured natural language suffer from several limitations both in eliciting and specifying privacy requirements. In our previous study, we proposed a structured natural-language approach called the “Privacy Criteria Method” (PCM), which demonstrates potential advantages over user stories. Our goal is to present a PCM evaluation that focused on the opinions of software practitioners from different companies on PCM’s ability to support the specification of privacy requirements and the quality of the privacy requirements specifications produced by these software practitioners. We conducted a multiple case study to evaluate PCM in four different industrial contexts. We gathered and analysed the opinions of 21 practitioners on PCM usage regarding Coverage, Applicability, Usefulness, and Scalability. Moreover, we assessed the syntactic and semantic quality of the PCM artifacts produced by these practitioners. PCM can aid developers in elaborating requirements specifications focused on privacy with good quality. The practitioners found PCM to be useful for their companies’ development processes. PCM is considered a promising method for specifying privacy requirements. Some slight extensions of PCM may be required to tailor the method to the characteristics of the company. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024.

Adversarial Machine Learning (AML) discusses the act of attacking and defending Machine Learning (ML) Models, an essential building block of Artificial Intelligence (AI). ML is applied in many software-intensive products and services and introduces new opportunities and security challenges. AI and ML will gain even more attention from the industry in the future, but threats caused by already-discovered attacks specifically targeting ML models are either overseen, ignored, or mishandled. Current AML research investigates attack and defense scenarios for ML in different industrial settings with a varying degree of maturity with regard to academic rigor and practical relevance. However, to the best of our knowledge, a synthesis of the state of academic rigor and practical relevance is missing. This literature study reviews studies in the area of AML in the context of industry, measuring and analyzing each study's rigor and relevance scores. Overall, all studies scored a high rigor score and a low relevance score, indicating that the studies are thoroughly designed and documented but miss the opportunity to include touch points relatable for practitioners. © 2024 The Author(s)

Systems that incorporate Machine Learning (ML) models, often referred to as ML-enabled systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited; this is especially true for activities surrounding ML model dissemination. [Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring ML life cycle phases. [Method] We conducted an international survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered a total of 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We analyzed contemporary practices using bootstrapping with confidence intervals and conducted qualitative analyses on the reported problems applying open and axial coding procedures. [Results] Practitioners perceive the model deployment and monitoring phases as relevant and difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and the selection of suitable metrics. [Conclusion] Our results help provide a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

Context: Regulatory acts are a challenging source when eliciting, interpreting, and analyzing requirements. Requirements engineers often need to involve legal experts who, however, may often not be available. This raises the need for approaches to regulatory Requirements Engineering (RE) covering and integrating both legal and engineering perspectives. Problem: Regulatory RE approaches need to capture and reflect both the elementary concepts and relationships from a legal perspective and their seamless transition to concepts used to specify software requirements. No existing approach considers explicating and managing legal domain knowledge and engineering-legal coordination. Method: We conducted focus group sessions with legal researchers to identify the core challenges to establishing a regulatory RE approach. Based on our findings, we developed a candidate solution and conducted a first conceptual validation to assess its feasibility. Results: We introduce the first version of our Artifact Model for Regulatory Requirements Engineering (AM4RRE) and its conceptual foundation. It provides a blueprint for applying legal (modelling) concepts and well-established RE concepts. Our initial results suggest that artifact-centric RE can be applied to managing legal domain knowledge and engineering-legal coordination. Conclusions: The focus groups that served as a basis for building our model and the results from the expert validation both strengthen our confidence that we already provide a valuable basis for systematically integrating legal concepts into RE. This overcomes contemporary challenges to regulatory RE and serves as a basis for exposure to critical discussions in the community before continuing with the development of tool-supported extensions and large-scale empirical evaluations in practice. © 2024 IEEE.

Systems that use Machine Learning (ML) have become commonplace for companies that want to improve their products and processes. Literature suggests that Requirements Engineering (RE) can help address many problems when engineering ML-enabled systems. However, the state of empirical evidence on how RE is applied in practice in the context of ML-enabled systems is mainly dominated by isolated case studies with limited generalizability. We conducted an international survey to gather practitioner insights into the status quo and problems of RE in ML-enabled systems. We gathered 188 complete responses from 25 countries. We conducted quantitative statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative analyses on the reported problems involving open and axial coding procedures. We found significant differences in RE practices within ML projects. For instance, (i) RE-related activities are mostly conducted by project leaders and data scientists, (ii) the prevalent requirements documentation format concerns interactive Notebooks, (iii) the main focus of non-functional requirements includes data quality, model reliability, and model explainability, and (iv) main challenges include managing customer expectations and aligning requirements with data. The qualitative analyses revealed that practitioners face problems related to lack of business domain understanding, unclear goals and requirements, low customer engagement, and communication issues. These results help to provide a better understanding of the adopted practices and of which problems exist in practical environments. We put forward the need to adapt further and disseminate RE-related practices for engineering ML-enabled systems. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

Recent studies have shown that many software developers do not have sufficient knowledge and understanding of how to develop a privacy-friendly system. This may become a challenge in developing systems complying with data protection laws. To address this issue, we investigated the factors that influence developers' decision-making when developing privacy-sensitive systems.

We conducted an empirical study by means of a survey with 109 practitioners. Our data analysis is based on the principles of social cognitive theory, which includes personal, behavioral, and external environmental factors.

We identified six personal, five behavioral, and five external environment factors that affect how developers make decisions regarding privacy, including confusion between privacy and security and reliance on informal practices and organizational support gaps. These findings contribute to understanding how practitioners and companies consider privacy, showing improvements in formal training and structured support over previous studies yet highlighting persistent challenges in consistent privacy integration. 

Quality attributes of software systems, also known as system qualities, such as performance, security, and scalability, continue to grow in importance in industrial practice. The evaluation of quality attributes is critical to software development since optimizing a software system’s core attributes can provide marketing advantage and set a product apart from its competitors. Many existing studies of unsuccessful development projects report that lack of quality attribute evaluation is often a contributing factor of project failure. Therefore, continuous quality attribute evaluation, throughout the development process, is needed to ensure customers’ expectations and demands are met.

Manual evaluation of software attributes is common in many software development companies, but it has proven to be insufficient in meeting the demands of rapid releases and high-quality expectations from customers. Automated practices have therefore gained widespread popularity as a solution to enhance efficiency, reduce costs, and increase accuracy compared to manual evaluation.

One way to automate the evaluation is using continuous integration (CI) environments. The CI environment provides several benefits, such as fast feedback on code quality, early detection of quality defects, and visualization of system quality trends. As such, these environments inherently offer organizations the opportunity to continuously monitor the quality of their software systems. However, an immature automation process can result in negative outcomes, such as cost and schedule overruns, slow feedback loops, and delayed releases.

To improve the evaluation process, prior studies have investigated different key areas, including knowledge, processes, tools, and metrics. While leveraging these areas can have a positive impact on quality evaluation, to the best of our knowledge, there is a lack of frameworks that link CI environment knowledge, metrics, and evolution together.

In this article, we aim to fill this gap by presenting the state-of-practice of using CI environments for the evaluation of quality attributes. This is achieved through an industrial study at four partner companies. Study results show that metrics acquired from CI components have a positive effect on evaluating quality requirements. Through analyzing these results, we propose a model by providing guidelines to mature existing CI environments that organizations can use for quality improvements.

As such, we claim the following contributions of this study:

A generic model of how CI environments contribute to quality attribute evaluation.

Empirical evidence that demonstrates how CI components can be used to produce data supporting the evaluation of quality attributes with metrics.

A model, derived from the study results, which provides decision support to evolve software quality evaluation through CI environments over time. © 2024 Owner/Author.

Context:Developing software-intensive products or services usually involves a plethora of software artefacts. Assets are artefacts intended to be used more than once and have value for organisations; examples include test cases, code, requirements, and documentation. During the development process, assets might degrade, affecting the effectiveness and efficiency of the development process. Therefore, assets are an investment that requires continuous management.

Identifying assets is the first step for their effective management. However, there is a lack of awareness of what assets and types of assets are common in software-developing organisations. Most types of assets are understudied, and their state of quality and how they degrade over time have not been well-understood.

Methods:We performed an analysis of secondary literature and a field study at five companies to investigate and identify assets to fill the gap in research. The results were analysed qualitatively and summarised in a taxonomy.

Results:We present the first comprehensive, structured, yet extendable taxonomy of assets, containing 57 types of assets.

Conclusions:The taxonomy serves as a foundation for identifying assets that are relevant for an organisation and enables the study of asset management and asset degradation concepts.