The performance of remotely rendered Virtual Reality (VR) is sensitive to temporal disturbances in communication channels. An earlier Quality of Experience (QoE) study of temporal impacts in the form of frame stalls has revealed difficulties with subjective disturbance ratings while performing a task in an interactive 6-degrees-of-freedom (DOF) VR environment. This study follows up on above observation by comparing QoE ratings in the presence and absence of a task. The exploratory findings show that the task-free subjective tests yield lower ratings compared to the subjective tests with task. This indicates that the participants became more sensitive to temporal impairments in the absence of a task. Also, the positive impact of reprojection on the QoE ratings decreased in the task-free environment. The simulator sickness results for individual symptoms were on similar low levels in both settings. The total score (TS) of sickness severity was higher after than before the subjective tests with task while the difference between the TS before and after the task-free subjective tests was insignificant, © 2023 IEEE.

The emerging 5G mobile networks are essential enablers for mobile virtual reality (VR) video streaming applications assuring high quality of experience (QoE) at the end-user. In addition, mobile edge computing brings computational resources closer to the user equipment (UE), which allows offloading computationally intensive processing. In this paper, we consider a network architecture for mobile VR video streaming applications consisting of a server that holds the VR video content, a mobile edge virtualization with prefetching (MVP) unit that handles the VR video packets, and a head-mounted display along with a buffer, which together serve as the UE. Several modulation and coding schemes with different rates are provided by the MVP unit to adaptively cope with the varying wireless link conditions to the UE and the state of the UE buffer. The UE buffer caches VR video packets as needed to compensate for the adaptive rates. A performance analysis is conducted in terms of blocking probability, throughput, queueing delay, and average packet error rate. To capture the effect of fading severity, the analytical expressions for these performance metrics are derived for Nakagami-m fading on the wireless link from the MVP unit to the UE. Numerical results show that the proposed system meets the network requirements needed to assure the QoE levels of different mobile VR video streaming applications. © 2022 by the authors. Licensee MDPI, Basel, Switzerland.

In this paper, a mobile virtual reality (VR) video streaming system is considered and assessed. The system comprises of a VR server, a VR center offering mobile edge computing along with a buffer, and a mobile head-mounted display (HMD). Markov-modulated processes are used to model the arrival and departure traffic at the VR center and the time-varying link quality of the wireless channel to the mobile HMD. As the Markovian arrival process at the VR center is independent of the link quality variations of the wireless channel, the VR center controls the delivery of VR video packets to the mobile HMD subject to the link quality of the wireless channel. The Markov models are used to derive expressions of queueing theoretical performance measures, i.e., throughput, queueing delay, and packet loss probability. Numerical results are provided to investigate performance trade-offs for real data traces. © 2024 IEEE.

Due to large advances in immersive media systems offering increased resolutions, frame rates, and dynamic ranges, applications such as watching 360° videos on head-mounted displays (HMDs) have become more popular. To assist the development of immersive media applications, subjective tests of 360° videos on HMDs are needed to obtain a ground truth on the quality as perceived by the end-users. The absolute category rating (ACR) method, standardized for subjective quality assessment of conventional videos, has also been used for assessing the subjective quality of 360° videos. Recently, the modified ACR (M-ACR) method has been proposed aiming at subjective quality assessment of 360° videos. Several subjective test campaigns have been conducted to assess the ACR and M-ACR method for evaluating 360° videos covering a wide range of resolutions, bit rates, and quantization parameters. The opinion scores given by the participants in these subjective tests are accumulated to mean option scores (MOSs) and confidence intervals. However, to the best of our knowledge, a detailed statistical analysis and comparison of the rating durations associated with casting opinion scores for 360° test videos in the ACR and M-ACR method has not been reported in the literature. In this paper, we therefore provide a rating duration analysis for subjective quality assessment of 360° videos for the ACR and M-ACR method. The results support experimental designs for 360° video quality assessment and include the following findings: 1) The mean rating duration is shorter for the ACR method, 2) The difference of the standard deviation of rating durations versus MOS between the ACR and M-ACR method is insignificant, 3) For a given mean rating duration, however, the standard deviation is smaller for the M-ACR method indicating that the quality rating task is easier to execute than with ACR. 4) The period to be allocated in subjective tests for rating 360° videos should exceed 5 s but can be kept below or set equal to 10 s. © 2020 IEEE.

In this paper, we conduct an analysis of variance (ANOVA) on opinion scores for MPEG-4 scalable video coding (SVC) and advanced video coding (AVC) standards. This work resorts on a publicly available database providing opinion scores from subjective experiments for several scenarios such as different bit rates and resolutions. In particular, ANOVA is used for statistical hypothesis testing to compare two or more sets of opinion scores instead of being constrained to pairs of sets of opinion scores as would be the case for t-tests. As the ANOVA tests of the different scenarios are performed for mean opinion scores (MOS), box plots are also provided in order to assess the distribution of the opinion scores around the median. It is shown that the opinion scores given to the reference videos in SVC and AVC for different resolutions are statistically significantly similar regardless of the content. Further, for the opinion scores of the considered database, the ANOVA tests support the hypothesis that AVC generally outperforms SVC although the performance difference may be less pronounced for higher bit rates. This work also shows that additional insights on the results of subjective experiments can be obtained by extending the analysis of opinion scores beyond MOS to ANOVA tests and box plots.

Watching 360° videos on head-mounted displays (HMDs) has become more popular in recent years. As such, subjective quality assessment of 360° videos on HMDs is needed to obtain a ground truth on the quality as perceived by users. Due to the lack of standardized methods, the absolute category rating (ACR) with hidden reference (ACR-HR) method for conventional videos has also been used for subjective quality assessment of 360° videos on HMDs. A modified ACR (M-ACR) method tailored for assessing 360° videos on HMDs has recently been proposed. In this paper, we compare the ACR-HR and M-ACR with hidden reference (M-ACR-HR) method regarding subjective quality assessment of 360° videos on HMDs. The comparison is conducted subject to participants' experience with watching immersive media on HMDs. The findings include: 1) Average rating times to cast opinion scores is lower for the ACR-HR method irrespective of participants' experience, 2) Mean opinion scores are similar for both methods irrespective of participants' experience, 3) The M-ACR-HR method is more reliable, especially with experts and participants with no experience, 4) Discomfort is significantly lower for the ACR-HR method, especially, for participants with no experience. © 2020 IEEE.

Subjective 360° video quality experiments provide the ground truth for developing related systems such that acceptable quality of experience is offered to the users. To obtain statistically meaningful insights from the data recorded in quality experiments, engaging a sufficiently large number of subjects is essential. In contrast to conventional videos, little has been reported on statistical measures that may guide the choice of the number of subjects required for 360° video quality experiments. In this paper, the standard deviation of opinion score (SOS) hypothesis, measuring the diversity among the opinion scores given by a panel of subjects, is used to reveal the number of subjects needed. It is shown that the number of subjects needed varies depending on a threshold on SOS characteristics in terms of diversity among opinion scores. © 2022 IEEE.

Immersive media such as virtual reality, augmented reality, and 360° video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360° video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent for each user. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360° video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360° video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community. © 2024. M. Elwardy et al., licensed to EAI. This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited. doi:10.4108/eetinis.v11i1.4323. All Rights Reserved.

The development of immersive media systems and services relies on subjective tests that provide ground truth on the quality of experience as perceived by humans. In this paper, we investigate the opinion score consistency through a repeated subjective 360° video quality assessment experiment. The test stimuli are presented on a head-mounted display with participants given the task to rate the video quality in three repeated sessions for both standing and seated viewing. The statistical analysis of the data from the subjective tests aims at revealing if participants change their rating behavior over time or keep their opinion scores given to the test stimuli in standing and seated viewing consistent in a statistical sense. The experimental results are reported in terms of histograms of opinion scores, skewness and kurtosis of opinion scores, mean opinion scores, standard deviation of opinion scores (SOS), SOS fitting functions, and analysis of variance tests. The statistical analysis supports the conjecture that each participant has its own but consistent rating behavior throughout the respective sessions for standing and seated viewing. The finding that the quality rating behavior of the individual participant does not fundamentally change over time may, e.g., assist in scheduling subjective tests under pandemic conditions where experimental campaigns may need to be stalled for an unknown period of time. © 2021 IEEE.

In this paper, objective perceptual video quality models are proposed that use spatial and temporal perceptual information differences for predicting video quality as perceived by human observers. Spatial perceptual information characterizes the complexity and temporal perceptual information quantifies the motion contained in a video. As such, differences in the spatial and temporal perceptual information of a reference video (original) and test video (processed) may be used to predict quality of videos that have undergone encoding, transmission, or other processing. In particular, several video quality prediction functions are derived using curve fitting along with training and validation on data from a publicly available annotated database. The obtained functions provide predicted mean opinion scores as a measure of perceptual quality subject to spatial and temporal perceptual information differences. The analysis of the video quality prediction performance of the proposed models shows that differences in spatial and temporal perceptual information can be used for objective video quality prediction. © 2019 IEEE.

The recent advances in immersive technologies have been essential in the development of a wide range of novel standalone and networked immersive media applications. The concepts of virtual reality, augmented reality, and mixed reality relate to different compositions of real and computer-generated virtual objects. In this context, 360° video streaming has become increasingly popular offering improved immersive experiences when viewed on a head-mounted display (HMD). An important component in the development of novel immersive media systems are subjective tests in which participants assess the quality of experience of representative test stimuli. In this paper, the annotated ACR360 dataset is presented which is publicly available on GitHub. The ACR360 dataset contains a wide range of psychophysical and psychophysiological data that was collected in Subjective tests on 360° video quality. The test stimuli were shown on an HMD and rated according to the absolute category rating (ACR) and modified ACR (MACR) methods. To support an easy exploration and utilization of the ACR360 dataset by the research community, its structure on GitHub is described and a comprehensive illustration of analysis options are provided for each data category. The ACR360 dataset may be used for conducting meta-analysis in combination with other datasets to improve precision and to pursue research questions that cannot be answered by an individual study. © 2023 IEEE.

Virtual reality (VR) has seen tremendous advances in head-mounted displays (HMDs), optics, media quality, and other improvements that facilitate immersive experiences. With the occurrence of new technologies like Cloud VR and networked VR video services, applications such as 360° video streaming are becoming more popular within the broader consumer markets. As a result, VR content is accessible to customers with rather different levels of experiences with immersive media, i.e., never, sometimes, or often use of VR. The question, therefore, arises to which degree simulator sickness is induced to viewers depending on their experiences with VR on HMDs. In this paper, simulator sickness is evaluated for 360° videos that were shown on an HTC Vive Pro HMD to participants having different levels of experience with VR on HMDs. The modified absolute category rating with hidden reference (M-ACR-HR) method was used in a subjective experiment for video quality assessment within two subsequent sessions along with a simulator sickness questionnaire (SSQ). A statistical analysis of the SSQ scores is performed to reveal the relationship between simulator sickness and participants' experiences with VR regarding: (1) Individual symptoms, (2) Pairwise comparison of symptoms, and (3) Symptom clusters of nausea, oculomotor, disorientation, and total score. It is shown that the simulator sickness symptoms, in general, are slightly or rarely perceived across the different experience levels for the selected 360° videos. The results indicate that the reported simulator sickness increases in the second session for participants that never used VR on HMDs. Sufficiently long breaks between sessions should therefore be accounted for in the M-ACR-HR method to avoid that simulator sickness influences quality rating. © 2020 IEEE.

The advances in 5G mobile networks are expected to enable immersive interconnected mobile multimedia systems. As humans are the final judges of the quality of immersive multimedia, it is essential to engage a suitable ground truth in the design of such systems. Databases annotated with results from subjective tests constitute such ground truth given as opinion scores, head movements, eye tracking data, psychophysiological data, and other data related to the viewers' behavior. On this basis, perception-based quality assessment of algorithms, systems, and services can be performed, and objective perceptual quality models can be developed. In this paper, a comprehensive survey of publicly available annotated 360-degree image and video databases is provided. The survey may guide the selection of ground truth on 360-degree images and videos to support quality assessment and modeling research. Further, the survey reveals the need for establishing new annotated databases that address the full range of subjective aspects of immersive multimedia. © 2019 IEEE.

In this paper, we study the impact that the level of experience of participants with immersive multimedia has on 360° video quality assessment. An extensive subjective experiment using the modified absolute category rating with hidden reference (M-ACR-HR) method was conducted. The opinion scores obtained from the participants in this experiment along with other sensor data support this study. The statistical analysis is performed in terms of screening participants' distribution of opinion scores for consistency, average rating time, box plots related to mean opinion scores (MOSs) and differential MOS (DMOS). Detailed results for MOS and DMOS are also provided for reference and test sequences covering a wide range of qualities. Major findings include: (1) Screening detects many outliers for participants with no experience, (2) Average rating times show that experts learn the quality assessment task fast, (3) Participants with no experience provide lower opinion scores in average, (4) DMOS may alleviate the bias in quality rating caused by different levels of experience. © 2019 IEEE.

This work focuses on post-analysis of performance results by means of Performance Analytics. The results to be post-analysed are provided by a Stochastic Fluid Flow Model (SFFM) of Virtual Reality (VR) streaming. Performance Analytics implies using the Machine Learning (ML) algorithm M5P for constructing model trees, which we examine amongst others for asymptotic behaviours and parameter impacts in both uni- and multivariate settings. We gain valuable insights into key parameters and related thresholds of importance for good VR streaming performance. © Springer Nature Switzerland AG 2020.

Virtual Reality (VR) applications put high demands on software and hardware in order to enable an immersive experience for the user and avoid causing simulator sickness. As soon as networks become part of the Motion-To-Photon (MTP) path between rendering and display, there is a risk for extraordinary delays that may impair Quality of Experience (QoE). This short paper provides an overview of latency measurements and models that are applicable to the MTP path, complemented by demands on user and network levels. It specifically reports on freeze duration measurements using a commercial TPCAST wireless VR solution, and identifies a corresponding stochastic model of the freeze length distribution, which may serve as disturbance model for VR QoE studies. © 2019 IEEE.

Augmented Reality (AR) and 3D printing have increasingly been used in archaeological and cultural heritage to make artifacts and environments accessible to the general public. This paper presents the case study of the Ljungaviken dog, an archaeological find of dog skeleton remains dated around 8000 years ago. The dog remains have been digitized using 3D scanning and displayed in an AR application. A physical replica has also been created with 3D printing. Both the AR application and the 3D printed copy have been shown in a temporary museum exhibition. In this paper, we present the visitors’ experience evaluation based on a study with 42 participants. Aspects being evaluated are related to the realism, enjoyment, and easiness of use of the AR application. Moreover, the two media are compared in terms of understanding, visual quality, and experience satisfaction. The results show an overall positive experience for both the display solutions, with slightly higher scores for the AR application in the comparison. When asked about overall preference, the participants reported similar results between both media. Due to issues of displaying fragile objects in a museum setting, as well as recent restrictions following pandemic closures and availability, the results presented in this paper show a positive alternative towards using digital artifacts to showcase our cultural heritage. © 2022, Springer Nature Switzerland AG.

In eye tracking data visualization, areas of interest (AOIs) are widely adopted to analyze specific regions of the stimulus. We propose a visual analytics tool that leverages panoptic segmentation to automatically divide the whole image or frame video in semantic AOIs. A set of AOI-based visualization techniques are available to analyze the fixation data based on these semantic AOIs. Moreover, we propose a modified version of radial transition graph visualizations adapted to the extracted semantic AOIs and a new visualization technique also based on radial transition graphs. Two application examples illustrate the potential of this approach and are used to discuss its usefulness and limitations.

Recent research in 3D shape analysis focuses on the study of visual attention on rendered 3D shapes investigating the impact of different factors such as material, illumination, and camera movements. In this paper, we analyze how the pose of a deformable shape affects visual attention. We describe an eye-tracking experiment that studied the influence of different poses of non-rigid 3D shapes on visual attention. The subjects free-viewed a set of 3D shapes rendered in different poses and from different camera views. The fixation maps obtained by the aggregated gaze data were projected onto the 3D shapes and compared at vertex level. The results indicate an impact of the pose for some of the tested shapes and also that view variation influences visual attention. The qualitative analysis of the 3D fixation maps shows high visual focus on the facial regions regardless of the pose, coherent with previous works. The visual attention variation between poses appears to correspond to geometric salient features and semantically salient parts linked to the action represented by the pose. © 2020, Vaclav Skala Union Agency. All rights reserved.

In the development of novel algorithms and techniques in virtual and augmented reality (VR/AR), it is crucial to take human visual perception into account. For example, when hardware resources are a restraining factor, the limitations of the human visual system can be exploited in the creation and evaluation of new effective techniques. Over the last decades, visual perception evaluation studies have become a vital part of the design, development, and evaluation of immersive computer graphics applications. This course aims at introducing the attendees to the basic concepts of visual perception applied to computer graphics and it offers an overview of recent perceptual evaluation studies that have been conducted with head-mounted displays (HMDs) in the context of VR and AR applications. During this course, we call attention to the latest published courses and surveys on visual perception applied to computer graphics and interaction techniques. Through an extensive search in the literature, we have identified six main areas in which recent visual perceptual evaluation studies have been focused on: Distance perception, avatar perception, image quality, interaction, motion perception, and cybersickness. Trends, main results, and open challenges are discussed for each area and accompanied with relevant references offering the attendees a wide introduction and perspective on the topic. © 2019 Owner/Author.

In recent decades, a growing number of museums have adopted digital media, both as an enhancement of exhibitions of real artifacts and as an alternative to traditional display methods. The digital acquisition of artifacts generates accurate 3D replicas that can be displayed via different digital media. With an increase in immersive technologies in the cultural heritage (CH) domain, it is common to see digital artifacts presented in Virtual Reality (VR) and Augmented Reality (AR). This paper presents two user studies conducted in different locations evaluating the use of an AR application in the portrayal of the Mjallby Crucifix artifact. This paper presents the overall results from both user studies evaluating and discussing the AR application on a number of different aspects on a 7-point Likert scale: (1) understanding the artifact, (2) ease of use, (3) object feeling part of reality, (4) perceived visual quality of the object, (5) overall satisfaction experience, and (6) willingness to download the AR application. The results have been compared between genders, age groups, and previous experience with AR. Potential benefits and disadvantages of AR experiences in the context of a museum exhibition were also gathered in free text from the visitors.

In this paper, we compare the impact that standing and seated viewing of 360 degrees videos on head-mounted displays has on subjective quality assessment. The statistical analysis of the data gathered in a pilot study is reported in terms of average rating times, mean opinion scores, and simulator sickness scores. The results indicate: (1) Average rating times consumed for 360 degrees video quality assessment are similar for standing and seated viewing, (2) Higher resolving power among different quality levels is obtained for seated viewing, (3) Simulator sickness is kept significantly lower when seated.

In this paper, we examine the sensitivity of the digital multimedia broadcasting (DMB) MPEG-2 transport stream (TS) format to transmission errors. To find the sensitivity of different parts of TS packets to transmission errors, each TS packet is divided into four cells, i.e., the first three cells comprising 48 bytes each and the last cell is of 44 bytes length. Bit errors are then introduced into these different parts of the TS packets. The sensitivity of DMB videos to transmission errors and their locations is assessed in terms of the following measures: 1) Number of decoder crashes; 2) Number of decodable videos; 3) Total number of decodable frames; and 4) Objective perceptual video quality of the decoded videos. The structural similarity index and visual information fidelity criterion are used as objective perceptual quality metrics. Simulations are performed on seven different DMB videos using various bit error rates. The results show that the first cell of the TS packets is highly sensitive to bit errors compared to the subsequent three cells, both in terms of spatial and temporal video quality. Further, the sensitivity decreases from Cell 1 to Cell 4 of a DMB TS packet. The error sensitivity analysis reported in this paper may guide the development of more reliable transmission systems for future DMB systems and services. Specifically, the insights gained from this study may support designing better error control schemes that take the sensitivity of different parts of DMB TS packets into consideration.

Virtual Reality (VR) is becoming increasingly popular, and wireless cable replacements unleash the user of VR Head Mounted Displays (HMD) from the rendering desktop computer. However, the price to pay for additional freedom of movement is a higher sensitivity of the wireless solution to temporal disturbances of both video frame and input traffic delivery, as compared to its wired counterpart. This paper reports on the development of a test-bed to be used for studying temporal delivery issues of both video frames and input traffic in a wireless VR environment, here using TPCAST with a HTC Vive headset. We provide a solution for monitoring and recording of traces of (1) video frame freezes as observed on the wireless VR headset, and (2) input traffic from the headset and hand controls to the rendering computer. So far, the test-bed illustrates the resilience of the underlying WirelesslID technology and TCP connections that carry the input traffic, and will be used in future studies of Quality of Experience (QoE) in wireless desktop VR.

A large portion of today's network traffic consists of streamed video of large variety, such as films, television shows, live-streamed games and recently omnidirectional videos. A common way of delivering video is by using Dynamic Adaptive Streaming over HTTP (DASH), or recently with encrypted HTTPS. Encrypted video streams disable the use of Quality of Service (QoS) systems that rely on knowledge of application-dependent data, such as video resolution and bit-rate. This could make it difficult for a party providing bandwidth to efficiently allocate resources and estimate customer satisfaction. An application-independent way of measuring video stream quality could be of interest for such a party. In this paper, we investigate encrypted streaming of omni-directional video via YouTube to a smartphone in a Google Cardboard VR-headset. We monitored such sessions, delivered via both WiFi and mobile networks, at different times of day, implying different levels of congestion, and characterised the network traffic by using the Coefficient of Throughput Variation (CoTV) as statistic. We observe that this statistic shows to be able to indicate whether a stream is stable or unstable, in terms of potential video playback freezes, when the DASH delivery strategy is used.

This paper shows the impact of bitrate settings on objective quality measures when streaming non-panoramic remote-rendered Virtual Reality (VR) images. Non-panoramic here refers to the images that are rendered and sent across the network, they only cover the viewport of each eye, respectively.

To determine the required bitrate of remote rendering for VR, we use a server that renders a 3D-scene, encodes the resulting images using the NVENC H.264 codec and transmits them to the client across a network. The client decodes the images and displays them in the VR headset. Objective full-reference quality measures are taken by comparing the image before encoding on the server to the same image after it has been decoded on the client. By altering the average bitrate setting of the encoder, we obtain objective quality scores as functions of bitrates. Furthermore, we study the impact of headset rotation speeds, since this will also have a large effect on image quality.

We determine an upper and lower bitrate limit based on headset rotation speeds. The lower limit is based on a speed close to the average human peak head-movement speed, 360°s. The upper limit is based on maximal peaks of 1080°s. Depending on the expected rotation speeds of the specific application, we determine that a total of 20--38Mbps should be used at resolution 2160×1200@90,fps, and 22--42Mbps at 2560×1440@60,fps. The recommendations are given with the assumption that the image is split in two and streamed in parallel, since this is how the tested prototype operates.

Rendering VR-content generally requires large image resolutions. This is both due to the display being positioned close to the eyes of the user and to the super-sampling typically used in VR. Due to the requirements of low latency and large resolutions in VR, remote rendering can be difficult to support at sufficient speeds in this medium.In this paper, we propose a method that can reduce the required resolution of non-panoramic VR images from a codec perspective. Because VR images are viewed close-up from within a headset with specific lenses, there are regions of the images that will remain unseen by the user. This unseen area is referred to as the Hidden-Area Mesh (HAM) and makes up 19% of the screen on the HTC Vive VR headset as one example. By remapping the image in a specific manner, we can cut out the HAM, reduce the resolution by the size of the mesh and thus reduce the amount of data that needs to be processed by encoder and decoder. Results from a prototype remote renderer show that by using the proposed Hidden-Area Mesh Remapping (HAMR), an implementation-dependent speed-up of 10-13% in encoding, 17-18% in decoding and 7-11% in total can be achieved while the negative impact on objective image quality in terms of SSIM and VMAF remains small. © 2021 IEEE.

All virtual reality (VR) systems have some inherent hand-controller latency even when operated locally. In remotely rendered VR, additional latency may be added due to the remote transmission of data, commonly conducted through shared low-capacity channels. Increased latency will negatively affect the performance of the human VR operator, but the level of detriment depends on the given task. This work quantifies the relations between aiming accuracy and hand-controller latency, virtual target speed, and the predictability of the target motion. The tested context involves a target that changes direction multiple times while moving in straight lines. The main conclusions are, given the tested context, first, that the predictability of target motion becomes significantly more important as latency and target speed increase. A significant difference in accuracy is generally observed at latencies beyond approximately 130 ms and at target speeds beyond approximately 3.5 degrees/s. Second, latency starts to significantly impact accuracy at roughly 90 ms and approximately 3.5 degrees/s if the target motion cannot be predicted. If it can, the numbers are approximately 130 ms and 12.7 degrees/s. Finally, reaction times are on average 190-200 ms when the target motion changes to a new and unpredictable direction.

Aiming is key for virtual reality (VR) interaction, and it is often done using VR controllers. Recent eye-tracking integrations in commercial VR head-mounted displays (HMDs) call for further research on usability and performance aspects to better determine possibilities and limitations. This paper presents a user study exploring gaze aiming in VR compared to a traditional controller in an “aim and shoot” task. Different speeds of targets and trajectories were studied. Qualitative data was gathered using the system usability scale (SUS) and cognitive load (NASA TLX) questionnaires. Results show a lower perceived cognitive load using gaze aiming and on par usability scale. Gaze aiming produced on par task duration but lower accuracy on most conditions. Lastly, the trajectory of the target significantly affected the orientation of the HMD in relation to the target’s location. The results show potential using gaze aiming in VR and motivate further research. © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Electrodermal activity (EDA) is considered to be an effective metric for measuring changes in the arousal level of people. In this paper, the phasic component of EDA data from players is analyzed in relation to their reported experience from a standardized questionnaire, when interacting with a couple of virtual reality games that featured two different input devices: the HTC Vive and Leap Motion controllers. Initial results show that there are no significant differences in the phasic component data, despite having significant differences in their respective player experience. Furthermore, no linear correlations are found between the phasic component data and the evaluated experience variables, with the only exception of negative affect which features a weak positive correlation. In conclusion, the phasic component of EDA data has here shown a limited correlation with player experience and should be further explored in combination with other psychophysiological signals.

Thanks to its effectiveness, electrodermal activity (EDA) has been previously included as an evaluation metric, within analyses of user experience. In this study, the phasic component of participants’ EDA data is examined in relation to their reported experiences when playing a set of virtual reality games, that featured the HTC Vive and Leap Motion controllers for input. Two models are used in the analysis of the phasic component: a deconvolution model and a convex optimization model. Despite having significant differences in their player experiences, results indicate that there are not many significant differences in the phasic component data. Even if some weak correlations were found, the majority of results show no linear correlations between the phasic component data and the reported experience variables. This shows that the phasic component of EDA data should be further investigated in conjunction with other psychophysiological signals because it has only recently demonstrated a weak link with player experience. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

An important aspect of virtual reality (VR) interfaces are novel natural user interactions (NUIs). The increased use of VR games requires the evaluation of novel interaction techniques that allow efficient manipulations of 3D elements using the hands of the player. Examples of VR devices that support these interactions include the HTC Vive controller and the Leap Motion sensor. This paper presents a quantitative and qualitative evaluation of player performance and experience in a controlled experiment with 20 volunteering participants. The experiment evaluated the HTC Vive controller and the Leap Motion sensor when manipulating 3D objects in two VR games. The first game was a Pentomino puzzle and the second game consisted of a ball-throwing task. Four interaction techniques (picking up, dropping, rotating, and throwing objects) were evaluated as part of the experiment. The number of user interactions with the Pentomino pieces, the number of ball throws, and game completion time were metrics used to analyze the player performance. A questionnaire was also used to evaluate the player experience regarding enjoyment, ease of use, sense of control and user preference. The overall results show that there was a significant decrease in player performance when using the Leap Motion sensor for the VR game tasks. Participants also reported that hand gestures with the Leap Motion sensor were not as reliable as the HTC Vive controller. However, the survey showed positive responses when using both technologies. The paper also offers ideas to keep exploring the capabilities of NUI techniques in the future. Copyright © 2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

The area of biofeedback interaction has grown over recent years, thanks to the release of more affordable and reliable sensor technology, and the accessibility offered by modern game development tools. This article presents a systematic literature review focusing on how different biofeedback interaction methods have been used for entertainment purposes in video games, between 2008 and 2020. It divides previous contributions in terms of a proposed interaction classification criteria and five different biofeedback methods (with a sixth category combining them): electroencephalography, electrocardiography, eye tracking, electrodermal activity, electromyography, and multi-modal interaction. The review describes the properties, sensor technologies, and the type of data gathered for every included biofeedback method, and presents their respective interaction techniques. It summarizes a set of opportunities and challenges for each included method, based on the results from previous work, and discusses these findings. It also analyzes how these interaction techniques are distributed between different common game genres. The review is beneficial for people interested in biofeedback methods and their potential use for novel interaction techniques in future video games. © 2021 ACM.

In this paper, we propose a viewing direction based least significant bit (LSB) data hiding method for 360◦ videos. The distributions of viewing direction frequency for latitude and longitude are used to control the amount of secret data to be hidden at the latitude, longitude, or both latitude and longitude of 360◦ videos. Normalized Gaussian mixture models mimicking the viewing behavior of humans are formulated to define data hiding weight functions for latitude, longitude, and both latitude and longitude. On this basis, analytical expressions for the capacity offered by the proposed method to hide secret data in 360◦ cover videos are derived. Numerical results for the capacity using different numbers of bit planes and popular 360◦ video resolutions for data hiding are provided. The fidelity of the proposed method is assessed in terms of the peak signal-to-noise ratio (PSNR), weighted-to-spherically uniform PSNR (WS-PSNR), and non-content-based perceptual PSNR (NCP-PSNR). The experimental results illustrate that NCP-PSNR returns the highest fidelity because it gives lower weights to the impact of LSB data hiding on fidelity outside the front regions near the equator. The visual quality of the proposed method as perceived by humans is assessed using the structural similarity (SSIM) index and the non-content-based perceptual SSIM (NCP-SSIM) index. The experimental results show that both SSIM-based metrics are able to account for the spatial perceptual information of different scenes while the PSNR-based fidelity metrics cannot exploit this information. Furthermore, NCP-SSIM reflects much better the impact of the proposed method on visual quality with respect to viewing directions compared to SSIM. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

In this paper, we propose a viewing direction weighted bit plane least significant bit (LSB) data hiding method for 360° videos. Normalized Gaussian mixture models (GMMs) of the distributions of the relative viewing direction frequency for latitude, longitude, and both latitude and longitude are used to describe the human viewing behavior of 360° videos on head-mounted displays (HMDs). Apart from formulating data hiding weight functions to control the amount of secret data to be hidden at given latitudes and longitudes, the GMMs are used in this work also to define viewing direction weighted bit plane functions. These viewing direction weighted bit plane functions specify the number of bit planes to be used for data hiding depending on the latitude and longitude. As humans pay more attention to the central front view of 360° videos when watched on HMDs, the number of engaged bit planes is decreased accordingly from the poles to the equator, and from the west and the east to the front view. A performance assessment of the proposed viewing direction weighted bit plane LSB data hiding method is conducted, showing the trade-offs between capacity and objective perceptual video quality. The results indicate that it is preferable to first utilize a given bit plane for data hiding in an entire 360° video frame before engaging the next higher bit plane subject to the viewing direction weighted bit plane functions. © 2021 IEEE.

In this paper, we propose a visual attention based least significant bit (LSB) data hiding method for 360° videos. In particular, information about the relative frequency of pixel access is used to control the amount of secret data to be hidden at the different latitudes of 360° videos. In this way, the typical behavior of users paying more attention to the equator region compared to the north and south poles when viewing 360° videos on a head-mounted display is taken into account. An analytical expression for capacity offered by this method to hide secret data in 360° cover videos is derived. A performance assessment of the visual attention based LSB data hiding method is conducted in terms of the peak signal-to-noise ratio (PSNR) and weighted-to-spherically-uniform PSNR (WS-PSNR) which both quantify the fidelity of a 360° stego-video with reference to the related 360° cover video. © 2020 IEEE.

In recent years, we have seen an increasing interest in immersive mobile multimedia applications and services. Advanced human-machine interfaces (HMIs) and related head-mounted displays (HMDs) support different levels of computer-generated virtual worlds. These worlds include mobile virtual reality (VR), mobile augmented reality (VR), and mobile mixed reality (MR) which are jointly captured under the term mobile extended reality (XR). In addition, the development of fifth generation (5G) mobile networks aims at catering for ultra-reliable low-latency communications such as mobile XR and the tactile Internet. In this context, 5G mobile networks are essential enablers to carry mobile XR beyond video gaming to the general consumer and industry markets. In this keynote lecture, the fundamentals of interconnected immersive mobile multimedia are reviewed, and opportunities, challenges, and enablers are discussed. An overview of immersive mobile multimedia from mobile video to mobile 360-degree video, mobile VR, mobile AR, mobile MR, and mobile XR is provided. Major components of interconnected immersive mobile multimedia will be discussed and application opportunities in different vertical industries are pointed out. Furthermore, challenges associated with the delivery of immersive mobile XR applications are shown through service integrity parameters. Several enablers for interconnected immersive mobile multimedia applications are highlighted including standardization activities.

In this paper, we provide a descriptive statistical analysis of subjective experiments in mobile imaging. Most of the statistical analyses of such experiments use condensed statistics such as mean opinion scores along with standard deviations. These statistics give only limited insights on the underlaying distribution of a data set. On the other hand, higher order statistics such as skewness and kurtosis provide additional information about a distribution but their qualitative interpretation is complicated. In order to alleviate these shortcomings, we advocate the use of box plots to efficiently visualize and compare distributions between different sets of image quality scores. Specifically, box plots serve the purposes of accounting for the diversity in image quality scores as well as supporting an intuitive qualitative interpretation of the results. It is shown by analysing two subjective experiments of the wireless imaging quality database that notched box plots are beneficial in quickly assessing the statistical significance among differences in such experiments. Further, box plots are used to identify those test material that received a wide diversity in image quality scores including outliers.

Immersive multimedia applications such as 360° videos and virtual reality (VR) live broadcasting are expected to receive increased popularity in the consumer markets. These applications provide three-dimensional spatial information and require to develop the related advanced wired and wireless ecosystems ranging from head-mounted displays (HMDs) and processors to network connectivity. To develop immersive multimedia systems that provide acceptable user experiences, it is essential to conduct subjective experiments that support assessing subjective video quality as perceived by the users. In this paper, we focus on the question of how many participants are needed in subjective experiments for 360° videos such that a statistically meaningful subjective quality assessment is supported. For this purpose, a subjective experiment for 360° videos has been conducted using the modified absolute category rating with hidden reference (M-ACR-HR) method. A statistical analysis of the opinion scores given by the participants to the visual stimuli shown on an HMD is provided. In particular, mean opinion scores (MOS), differential MOS (DMOS), and the associated margin of error (MoE) and differential MoE are obtained. It is shown that 21 to 26 participants would be sufficient to drive the differential MoE of the DMOS below the given thresholds of ϵ =1% and ϵ=0.75%, respectively. © 2019 IEEE.

Wireless virtual reality (VR) offers a seamless user experience but has to cope with higher sensitivity to temporal impairments induced on the wireless link. Apart from bandwidth dynamics and latency, video freezes and their lengths are important temporal performance indicators that impact on the quality of experience (QoE) of networked VR applications and services. This paper reports an experimental study that focuses on the VR video frame freeze length characteristics of a wireless VR solution. A comprehensive measurement campaign using a commercial TPCAST wireless VR solution with an HTC Vive head-mounted display was conducted to obtain real VR video traces. The number of detected freezes and freeze intensities are reported both accumulated over four room quadrants as well as for each of the four quadrants subject to six transmitter-receiver distances. The statistical analysis of the VR video traces of the different experiments includes histograms of the freeze lengths and cumulative complementary histograms of the freeze length. The results of this analysis offer insights into the density of the underlying distributions of the measured data, illustrate the impact of the room topology on the freeze characteristics, and suggest the statistical modeling of the freeze characteristics as exponential and geometric distributions. The statistical models of the freeze characteristics may be included in wireless VR simulators supporting the development of physical layer, medium access layer, and higher layer functionalities. They also may serve as network-disturbance models for VR QoE studies, e.g., generating realistic freeze events in wireless VR stimuli. © 2022 by the authors. Licensee MDPI, Basel, Switzerland.