The experimental setting of Human Mobile Computer Interaction (HCI) studies is moving from the controlled laboratory to the user’s daily-life environments, while employing the users’ own smartphones. These studies are challenging for both new and expert researchers in human subject studies in the HCI field. Within the last three years, we conducted three different smartphone- based user studies. From these studies, we have derived key challenges that we successfully overcame during their execution. In this paper, we present the outcomes and explain the adopted solutions for the challenges identified in the design, development and execution, and data analysis phases during the user studies. Our goal is to give newcomers and junior researchers a practical view on our conducted studies, and help practitioners to reflect on their own studies and possibly apply the proposed solutions.

This report documents the program and the outcomes of Dagstuhl Seminar 12181 "Quality of Experience: From User Perception to Instrumental Metrics". As follow-up of the Dagstuhl Seminar 09192 "From Quality of Service to Quality of Experience", it focused on the further development of an agreed definition of the term Quality of Experience (QoE) in collaboration with the COST Action IC1003 "Qualinet", as well as inventories of possibilities to measure QoE (beyond the usual user polls) and to exploit feedback between users and systems that reflects QoE issues. The report furthermore describes the mode of work throughout the seminar, with focus on personal statements by the participants, results of the group works, and open challenges.

Mobile handheld devices have become friends in people’s daily lives. Frequent usage of available applications, especially video streaming, causes exponential growth in mobile IP traffic. Service providers and application developers need to know the tradeoff between the end-to-end (e2e) performance and cost since, not fully met expectations of customers from those applications cause reduced usage of services, revenue, and growth in the churn rate. The user-centric approach, which involves users into the assessment of the performance of a particular service or application, has become important within the inter-disciplinary research field Quality of Experience (QoE). The ultimate goal is to obtain simplified QoE models on particular applications based on the underlying network-based performance metrics as well as other non-technical metrics related to the end-user. Android smartphones that use open-source code and well-documented Application Programming Interfaces (API), facilitate researchers to do low-level and network-based performance analysis on end-user mobile devices while considering user feedback. In this thesis, the influential factors for Android smartphone-based QoE are studied. The relation between the quantified user-perceived QoE metric, i.e., Mean Opinion Score (MOS), and the artifacts in real-time video streaming such as blockiness and jerkiness caused by network-level metrics, e.g., Packet Delay Variation (PDV), Maximal Burst Size (MBS), and video bit rate are identified. Challenges in assessing the user-perceived QoE of video with the focus on memory effects are discussed. The relation between the objective metric of user reaction time and the user-perceived QoE is presented. Furthermore, different methods to assess end-user-perceived QoE such as Day Reconstruction Method (DRM), Experience Sampling Method (ESM), and preliminary online survey are described. Further influential factors, e.g., context, user routines, user lifestyle, and Quality of Service (QoS) metrics such as Round Trip Time (RTT) and Server Response Time (SRT), are studied. The thesis is concluded with preliminary findings that relate the instantaneous total power consumption to the jerkiness of a real-time video stream with evidences such as stalling events.

Smartphones have become crucial enablers for users to exploit online services such as learning, leisure, communicating, and socializing. The user-perceived quality of applications and services is an important factor to consider, in order to achieve lean resource management, to prevent user churn and revenue depletion of service or network providers. This is often studied within the scope of Quality of Experience (QoE), which has attracted researchers both in academia and industry. The objective of this thesis is to study the most important factors influencing QoE on smartphones and synthesize solutions for intervention. The temporal impairments during a real-time energy-hungry video streaming are studied. The aim is to quantify the influence of temporal impairments on the user-perceived video QoE at the network and application level together with energy measurements, and also to propose solutions to reduce smartphone energy consumption without degrading the user’s QoE on the smartphone for both user-interactive, e.g., video, and non-interactive cases. QoE measurements on smartphones are performed throughout in-the-wild user studies. A set of quantitative Quality of Experience (QoE) assessment tools are implemented and deployed for automatic data logging at the network- and application-level. Online momentary survey, Experience Sampling Method (ESM) software, and Day Reconstruction Method (DRM) along weekly face-to-face user interviews are employed. The subjective QoE is obtained through qualitative feedback including Mean Opinion Score (MOS) as well as in-situ indications of poor experiences by users. Additionally, energy measurements on smartphones are conducted in controlled-lab environment with the Monsoon device. The QoE of smartphone applications and services perceived by users depends on many factors including anomalies in the network, application, and also the energy consumption. At the network-level, high packet delay variation causes long video freezes that eventually impact negatively the end-user perceived quality. The freezes can be quantified as large time gaps in-between the displayed pictures during a video stream at the application-level. We show that the inter-picture time in cellular-based video stream can be represented via two-state exponential ON/OFF models. We show models representing the non-linear relationship between the QoE and the mean inter-picture time. It is shown that energy measurements help to reveal the temporal impairments in video stream enabling energy consumption as a QoE indicator. Next, energy waste and saving during temporal impairments are identified. Additionally, other video streaming use cases, e.g., “download first and watch later”, are studied and appropriate energy-saving download scheduling mechanisms are recommended. The possibility for decreasing energy consumption when the smartphone screen is OFF, while maintaining QoE, is revealed. We first show exponential models to represent user’s interaction with smartphone, then propose a NyxEnergySaver software, to control the cellular network interface in a personalized manner to save smartphone energy. According to our findings, more than 30% smartphone energy can be saved without impacting the user-perceived QoE.

For service and mobile operators, it is importantto monitor and keep high user engagement levels. Qualityof Experience (QoE) on video streaming applications is animportant engagement measure for video consumer customers.In this paper, video QoE (with the focus on stalling events)is studied from network, application, and energy perspectiveswith various instrumentations on a smartphone. This enablesthe understanding of inter-relation between the perspectives andalso how they influence the video QoE. Results show that packetdelay variation and the maximal burst size in the network level;inter-picture time (picture delay) in the application layer; andalso fluctuations in the energy consumptions are strong indicatorsfor QoE. We show, via extensive QoE and energy measurementson smartphones that, based on the choice of streaming protocol,energy consumption can be reduced or increased in the case ofstalling events during a video stream.

The smartphone usage nearly tripled 2011 according to Cisco Virtual Networking Index. There is a high demand of energy for using popular mobile applications, which run on smartphones with limited battery life. Video streaming applications are widely used on mobile devices, and their high power consumption exhibits high variance during a live streaming session, due to varying conditions on network and application levels. Recent studies focus on the averaged power consumption statistics, while there is lack of observation on the fluctuations of the instantaneous total power consumption of the smart- phones. Network based applications consume power at all layers of the communication stack, and any fluctuation in the total power consumption during a video streaming can reveal a possible misbehaviour such as a stalling event. Until now, these events are investigated in Quality of Experience (QoE) studies through installation of high-energy demanding and hard-to-deploy network measurement tools on users’ mobile devices. In this paper, we demonstrate an ex- periment, where a user experiences a stalling event on the smartphone and observes the live instantaneous power consumption values through Mobile Power Monitoring Tool (MPMT) and Software Visualisa- tion Tool (SVT), simultaneously. We confer that the instantaneous total power consumption likely reveals the misbehaviours such as stalls during a video play- out in live video streaming on smartphones that can facilitate energy efficient QoE studies.

Subjective performance of smartphone-based high bandwidth- and energy-demanding applications and services such as video streaming, are highly influenced by the temporal impairments perceived by the user at the user interface; and the application's energy consumption patterns. Therefore, we study the influence of the anomalies detected by objective measurements from the user interface and the network-level, on the power consumption during video streaming on the smartphone. In this paper, we study the inter-frame time metric, i.e., the time gap between two consecutive displayed pictures at the user interface; the inter-packet time and the initial signaling duration at the network-level; and the instantaneous power consumption at the power supply of the smartphone. We conduct experiments on the VLC media player, while streaming video via local-storage; via 3G using RTSP protocol; and via 3G using HTTP protocol. We show that the anomalies detected at the instantaneous power consumption reveals the anomalies at the user interface and the network-level.

The usage of network-demanding applications is growing rapidly such as video streaming on mobile terminals. However, network and/or service providers might not guarantee the perceived quality for video streaming that demands high packet transmission rate. In order to satisfy the user expectations and to minimize user churn, it is important for network operators to infer the end-user perceived quality in video streaming. Today, the most reliable method to obtain end-user perceived quality is through subjective tests, and the preferred location is the user interface as it is the closest point of application to the end-user. The end-user perceived quality on video streaming is highly influenced by occasional freezes; technically the extraordinary time gaps between two consecutive pictures that are displayed to the user, i.e., high inter-picture time. In this paper, we present a QoE instrumentation for video streaming, VLQoE. We added functionality to the VLC player to record a set of metrics from the user interface, application-level, network-level, and from the available sensors of the device. To the best of our knowledge, VLQoE is the first tool of its kind that can be used in user experiments for video streaming. By using the tool, we present a two state model based on the inter-picture time, for the HTTP- and RTSP-based video streaming via 3.5G. Next, we studied the influence of inter-picture time on the user perceived quality through out a user study. We investigated the minimum user perceived inter-picture time, and the user response time.

Evaluating video Quality of Experience (QoE) on a mobile phone has not yet been studied much. It is common that the data collected through user studies in mobile platform involves high fluctuation of user ratings without obvious reasons related to variation in network level. User disparity, user's various intermediate or previous experiences, video bitrate, and the objective measure of criticality are a few of the reasons that need to be identified with a well-designed user experiment. We present an experiment procedure to understand better the perceived quality of video in mobile platform. First, we investigate the reliability of the data, and identify unreliable users. Then, we investigate the psychological influence factors of previous experiences on the recent perceived quality known as the memory effect, and the influence of the bitrate on the time it takes for users to react and evaluate the video quality, i.e., user response time.

Practitioners on the area of mobile application development usually rely on set of app-related success factors, the majority of which are directly related to their economical/business profit (e.g., number of downloads, or the in-app purchases revenue). However, gathering also the user-related success factors, that explain the reasons why users choose, download, and install apps as well as the user-related failure factors that explain the reasons why users delete apps, might help practitioners understand how to improve the market impact of their apps. The objectives were to: identify (i) the reasons why users choose and installing mobile apps from app stores; (ii) the reasons why users uninstall the apps. A questionnaire-based survey involving 121 users from 26 different countries was conducted. © Springer International Publishing AG 2017.

One of the most influencing factors on the overall end-user perceived quality from applications and services, i.e., QoE, running on the smartphones is their limited battery life. Particular cloud-based applications/services on the smartphone with a constrained battery life might consume high energy even when the smartphone is in screen-OFF state. The cellular radio module of the smartphone is one of the most power-consuming components, which depends on the running applications' information polling characteristics that eventually cause the radio module to toggle occasionally between the cellular data energy states even during a sleep state. In this paper, we investigate the energy consumption of a set of applications that tend to retain up-to-date information via aggressive polling patterns. We show that limiting the network traffic and increasing the resource utilization efficiency amongst the applications and services can highly reduce the total energy consumption. We control the network activity of a smartphone with different cellular data-enabled and data-disabled durations at the screen-OFF state. First, we run controlled-lab energy measurements to have a ground truth on the power consumption patterns of a set of cloud-based popular applications/services; and next we conduct a subjective study with our proposed solution (ExpCO2), to understand first the user behaviour on the smartphone and then present how the reduced polling intervals of applications and notifications influence the end-user perceived quality. We indicate that ExpCO2 has a potential to save energy.

Increasingly, we use mobile applications and services in our daily life activities, to support our needs for information, communication or leisure. However, user acceptance of a mobile application depends on at least two conditions; the application’s perceived experience and the appropriateness of the application to the user’s context and needs. Yet, we have a weak understanding of a mobile user’s Quality of Experience (QoE) and the factors influencing it. This paper presents 4 week long, 29 Android phone users study, where we collected both QoE and underlying network’s Quality of Service (QoS) measures through a combination of user, application and network data on the user’s phones. We aimed to derive and improve the understanding of users’ QoE for a set of widely used mobile applications in users’ natural environments and different daily context. We present data acquired in the study and discuss implications for mobile applications design.

The most energy-consuming applications in battery life-constrained smartphones are the ones that comprise data transmission, especially via the 3G interface. Scheduling download activities on smartphones is especially necessary, if there are multiple asynchronous downloads scattered over a long duration. The latter scenario highly increases the energy consumption of smartphones. In this paper, we investigate energy consumption with the focus on file downloading while scheduling multiple file downloads in two scenarios: serialized and parallel. We repeat the experiments on a single smartphone via its 3G and also via WiFi tethering via another smartphone. We assess the performance of the two scenarios via measurement of power consumption and corresponding download duration in a realistic environment.