Context. Companies continuously explore their software systems to acquire evidence for software evolution, such as bugs in the system and new functional or quality requirements. So far, managers have made decisions about software evolution based on evidence gathered from interpreting user feedback and monitoring data collected separately from software in use. These evidence-collection processes are usually unmethodical, lack a systematic guide, and have practical issues. This lack of a systematic approach leaves unexploited opportunities for detecting evidence for system evolution. Objective. The main research objective is to improve evidence collection from software in use and guide software practitioners in decision-making about system evolution. Understanding useful approaches to collect user feedback and monitoring data, two important sources of evidence, and combining them are key objectives as well. Method. We proposed a method for gathering evidence from software in use (GESU) using design-science research. We designed the method over three iterations and validated it in the European case studies FI-Start, Supersede, and Wise-IoT. To acquire knowledge for the design, we conducted further research using surveys and systematic mapping methods. Results. The results show that GESU is not only successful in industrial environments but also yields new evidence for software evolution by bringing user feedback and monitoring data together. This combination helps software practitioners improve their understanding of end-user needs and system drawbacks, ultimately supporting continuous requirements elicitation and product evolution. GESU suggests monitoring a software system based on its goals to filter relevant data (i.e., goal-driven monitoring) and gathering user feedback when the system requests feedback about the software in use (i.e., system-triggered user feedback). The system identifies interesting situations of system use and issues automated requests for user feedback to interpret the evidence from user perspectives. We justified using goal-driven monitoring and system-triggered user feedback with complementary findings of the thesis. That showed the goals and characteristics of software systems constrain monitoring data. We thus narrowed the monitoring and observational focus on data aligned with goals instead of a massive amount of potentially useless data. Finally, we found that requesting feedback from users with a simple feedback form is a useful approach for motivating users to provide feedback. Conclusion. Combining user feedback and monitoring data is helpful to acquire insights into the success of a software system and guide decision-making regarding its evolution. This work can be extended in the future by implementing an adaptive system for gathering evidence from combined user feedback and monitoring data

Runtime monitoring and assessment of software products, features, and requirements allow product managers and requirement engineers to verify the implemented features or requirements, and validate the user acceptance. Gaining insight into software quality and impact of the quality on user facilitates interpretation of quality against users' acceptance and vice versa. The insight also expedites root cause analysis and fast evolution in the case of threatening the health and sustainability of the software. Several studies have proposed automated monitoring solutions and assessment, however, none of the studies introduces a solution for a joint assessment of software quality and quality impact on users. In this research, we study the relation between software quality and the impact of quality on Quality of Experience (QoE) of users to support the assessment of software products, features, and requirements. We propose a Quality-Impact assessment method based on a joint analysis of software quality and user feedback. As an application of the proposed method in requirement engineering, the joint analysis guides verification and validation of functional and quality requirements as well as capturing new requirements. The study follows a design science approach to design Quality-Impact method artifact. The Quality-Impact method has been already designed and validated in the first design cycle. However, next design cycles will contribute to clarify problems of the initial design, refine and validate the proposed method. This paper presents the concluded results and explains future studies for the follow up of the Ph.D. research.

Runtime requirement monitoring is used for verification and validation of implemented requirements. To monitor the requirements in runtime; we propose a "QoE probe" tool, a mobile application integrated through an API, to collect usage logs as well as users’ Quality of Experience (QoE) in the form of user feedback. The analysis of the collected data guides requirement monitoring of functional and non-functional requirements as well as capturing new requirements.

Evolution of a software product is inevitable as product context changes and the product gradually becomes less useful if it is not adapted. Planning is a basis to evolve a software product. The product manager, who carries responsibilities of planning, requires but does not always have access to high-quality information for making the best possible planning decisions. The current study aims to understand whether and when analytics are valuable for product planning and how they can be interpreted to a software product plan. The study was designed with an interview-based survey methodology approach through 17 in-depth semi-structured interviews with product managers. Based on results from qualitative analysis of the interviews, we defined an analytics-based model. The model shows that analytics have potentials to support the interpretation of product goals while is constrained by both product characteristics and product goals. The model implies how to use analytics for a good support of product planning evolution.

Quality requirements, an important class of non functional requirements, are inherently difficult to elicit. Particularly challenging is the definition of good-enough quality. The problem cannot be avoided though, because hitting the right quality level is critical. Too little quality leads to churn for the software product. Excessive quality generates unnecessary cost and drains the resources of the operating platform. To address this problem, we propose to elicit the specific relationships between software quality levels and their impacts for given quality attributes and stakeholders. An understanding of each such relationship can then be used to specify the right level of quality by deciding about acceptable impacts. The quality-impact relationships can be used to design and dimension a software system appropriately and, in a second step, to develop service level agreements that allow re-use of the obtained knowledge of good-enough quality. This paper describes an approach to elicit such quality-impact relationships and to use them for specifying quality requirements. The approach has been applied with user representatives in requirements workshops and used for determining Quality of Service (QoS) requirements based the involved users’ Quality of Experience (QoE). The paper describes the approach in detail and reports early experiences from applying the approach. Index Terms-Requirement elicitation, quality attributes, non-functional requirements, quality of experience (QoE), quality of service (QoS).

Companies are interested in knowing how users experience and perceive their products. Quality of Experience (QoE) is a measurement that is used to assess the degree of delight or annoyance in experiencing a software product. To assess QoE, we have used a feedback tool integrated into a software product to ask users about their QoE ratings and to obtain information about their rationales for good or bad QoEs. It is known that requests for feedback may disturb users; however, little is known about the subjective reasoning behind this disturbance or about whether this disturbance negatively affects the QoE of the software product for which the feedback is sought. In this paper, we present a mixed qualitative-quantitative study with 35 subjects that explore the relationship between feedback requests and QoE. The subjects experienced a requirement-modeling mobile product, which was integrated with a feedback tool. During and at the end of the experience, we collected the users' perceptions of the product and the feedback requests. Based on the users' rational for being disturbed by the feedback requests, such as "early feedback," "interruptive requests," "frequent requests," and "apparently inappropriate content," we modeled feedback requests. The model defines feedback requests using a set of five-tuple variables: "task," "timing" of the task for issuing the feedback requests, user's "expertise-phase" with the product, the "frequency" of feedback requests about the task, and the "content" of the feedback request. Configuration of these parameters might drive the participants' perceived disturbances. We also found that the disturbances generated by triggering user feedback requests have negligible impacts on the QoE of software products. These results imply that software product vendors may trust users' feedback even when the feedback requests disturb the users.

To create value with a software ecosystem (SECO), a platform owner has to ensure that the SECO is healthy and sustainable. Key Performance Indicators (KPI) are used to assess whether and how well such objectives are met and what the platform owner can do to improve. This paper gives an overview of existing research on KPI-based SECO assessment using a systematic mapping of research publications. The study identified 34 relevant publications for which KPI research and KPI practice were extracted and mapped. It describes the strengths and gaps of the research published so far, and describes what KPI are measured, analyzed, and used for decision-making from the researcher's point of view. For the researcher, the maps thus capture stateof- knowledge and can be used to plan further research. For practitioners, the generated map points to studies that describe how to use KPI for managing of a SECO.

SaaS cloud computing, in contrast to packaged products, enables permanent contact between users of a software product and the product-owning company. When planning the development and evolution of a software product, a product manager depends on reliable information about feature attractiveness. So far, planning decisions were based on stakeholder opinion and the customer's willingness to buy. Whether or not a feature actually is used was out of consideration. Analytics that measure the interaction between users and the SaaS gives product managers unprecedented access to information about product usage. To understand whether and how SaaS analytics can be used for product planning decision, we performed 17 in-depth interviews with experienced managers of SaaS products and analyzed the results analyzed with a mixed-method strategy. The empirical results characterize the relevance of a broad range of analytics for product planning decisions, and the strengths and limitations of an analytics-based product planning approach.

Collecting and using user feedback as a method to support requirements engineering, might undermine user rights. This becomes apparent when looking at related areas, e.g., research in user experience, where collecting user feedback also plays an important role. In such settings, researchers need to ensure that the stakeholders' rights and integrity are respected. This paper identifies and discusses some of the ethical challenges and issues a researcher can face, using an example case. Focusing on user feedback, this case can serve as an example for CrowdRE, i.e. several of our findings might apply to CrowdRE in general. However, further research is needed as our work mainly reflects the challenges experienced by the authors of this paper. © 2017 IEEE.

Shared understanding of requirements between stakeholders and the development team is a critical success factor for requirements engineering. Workshops are an effective means for achieving such shared understanding since stakeholders and team representatives can meet and discuss what a planned software system should be and how it should support achieving stakeholder goals. However, some important intended recipients of the requirements are often not present in such workshops: the developers. Thus, they cannot benefit from the in-depth understanding of the requirements and of the rationales for these requirements that develops during the workshops. The simple handover of a requirements specification hardly compensates the rich requirements understanding that is needed for the development of an acceptable system. To compensate the lack of presence in a requirements workshop, we propose to record that requirements workshop on video. If workshop participants agree to be recorded, a video is relatively simple to create and is able to capture much more aspects about requirements and rationales than a specification document. This paper presents the workshop video technique and a phenomenological evaluation of its use for requirements communication from the perspective of software developers. The results show how the technique was appreciated by observers of the video, present positive and negative feedbacks from the observers, and lead to recommendations for implementing the technique in practice.

Context: Electroencephalograms (EEG) have been used in a few cases in the context of software engineering (SE). EEGs allow capturing emotions and cognitive functioning. Such human factors have already shown to be important to understand software engineering tasks. Therefore, it is essential to gain experience in the community to utilize EEG as a research tool. Objective: To report experiences of using EEG in the context of a software engineering education (review of master theses proposals). We provide our reflections and lessons learned of (1) how to plan an EEG study, (2) how to conduct and execute (e.g., tools), (3) how to analyze. Method: We carried out an experiment using an EEG headset to measure the participants’ attention rate. The experiment task includes reviewing three master thesis project plans. Results: We describe how we evolved our understanding of experimentation practices to collect and analyze psychological and cognitive data. We also provide a set of lessons learned regarding the application of EEG technology for research. Conclusions: We believe that that EEG could benefit software engineering research to collect cognitive information under certain conditions. The lessons learned reported here should be used as inputs for future experiments in software engineering, where human aspects are of interest. © 2019 Association for Computing Machinery.

Context: Software evolution ensures that software systems in use stay up to date and provide value for end-users. However, it is challenging for requirements engineers to continuously elicit needs for systems used by heterogeneous end-users who are out of organisational reach. Objective: We aim at supporting continuous requirements elicitation by combining user feedback and usage monitoring. Online feedback mechanisms enable end-users to remotely communicate problems, experiences, and opinions, while monitoring provides valuable information about runtime events. It is argued that bringing both information sources together can help requirements engineers to understand end-user needs better. Method/Tool: We present FAME, a framework for the combined and simultaneous collection of feedback and monitoring data in web and mobile contexts to support continuous requirements elicitation. In addition to a detailed discussion of our technical solution, we present the first evidence that FAME can be successfully introduced in real-world contexts. Therefore, we deployed FAME in a web application of a German small and medium-sized enterprise (SME) to collect user feedback and usage data. Results/Conclusion: Our results suggest that FAME not only can be successfully used in industrial environments but that bringing feedback and monitoring data together helps the SME to improve their understanding of end-user needs, ultimately supporting continuous requirements elicitation. © 2018 IEEE.

End-user feedback is becoming more important for the evolution of software systems. There exist various communication channels for end-users (app stores, social networks) which allow them to express their experiences and requirements regarding a software application. End-users communicate a large amount of feedback via these channels which leads to open issues regarding the use of end-user feedback for software development, maintenance and evolution. This includes investigating how to identify relevant feedback scattered across different feedback channels and how to determine the priority of the feedback issues communicated. In this research preview paper, we discuss ideas for enduser driven feedback prioritization. © Copyright 2017 for this paper by its authors.

Feedback communication channels allow end-users to express their needs, which can be considered in software development and evolution. Although feedback gathering and analysis have been identified as an important topic and several researchers have started their investigation, information is scarce on how software companies currently elicit end-user feedback. In this study, we explore the experiences of software companies with respect to feedback gathering. The results of a case study and online survey indicate two sides of the same coin: On the one hand, most software companies are aware of the relevance of end-user feedback for software evolution and provide feedback channels, which allow end-users to communicate their needs and problems. On the other hand, the quantity and quality of the feedback received varies. We conclude that software companies still do not fully exploit the potential of end-user feedback for software development and evolution. © 2017 IEEE.

Crowd-based requirements engineering (CrowdRE) is promising to derive requirements by gathering and analyzing information from the crowd. Setting up CrowdRE in practice seems challenging, although first solutions to support CrowdRE exist. In this paper, we report on a German software company's experience on crowd involvement by using feedback communication channels and a monitoring solution for user-event data. In our case study, we identified several problem areas that a software company is confronted with to setup an environment for gathering requirements from the crowd. We conclude that a CrowdRE process cannot be implemented ad-hoc and that future work is needed to create and analyze a continuous feedback and monitoring data stream. © 2017 IEEE.

[Context and motivation] To validate developers’ ideas of what users might want and to understand user needs, it has been proposed to collect and combine system monitoring with user feedback. [Question/problem] So far, the monitoring data and feedback have been collected passively, hoping for the users to get active when problems emerge. This approach leaves unexplored opportunities for system improvement when users are also passive or do not know that they are invited to offer feedback. [Principal ideas/results] In this paper, we show how we have used goal monitors to identify interesting situations of system use and let a system autonomously elicit user feedback in these situations. We have used a monitor to detect interesting situations in the use of a system and issued automated requests for user feedback to interpret the monitoring observations from the users’ perspectives. [Contribution] The paper describes the implementation of our approach in a Smart City system and reports our results and experiences. It shows that combining system monitoring with proactive, autonomous feedback collection was useful and surfaced knowledge of system use that was relevant for system maintenance and evolution. The results were helpful for the city to adapt and improve the Smart City application and to maintain their internet-of-things deployment of sensors.