In the rapidly evolving virtual reality (VR) field, assessing video quality on head-mounted displays (HMDs) for 360° videos presents unique opportunities and challenges. As immersive multimedia becomes increasingly widespread, understanding how users perceive and evaluate the quality of 360° videos is essential. This thesis investigates the subjective quality assessment tests of 360° videos, examining how participants' VR experiences, viewing conditions, and exploration behaviors shape perceptions of quality. This thesis aims to perform subjective quality assessment tests to study and understand how participants perceive the quality of 360° videos on an HMD. The thesis starts with an extended summary of the field of subjective quality assessment for 360° videos, followed by eight key publications, and unfolds into three main parts.

The first part of the thesis focuses on data collection to establish ground truth. It includes a comprehensive survey of annotated 360° images and videos datasets related to subjective quality assessment. It also presents a set of datasets collected specifically for subjective quality assessment tests for 360° videos with different test methods and viewing conditions conducted as part of the research. The second part of the thesis investigates how varying levels of VR experience affect participants' video quality assessments. It compares two test methods, the absolute category rating (ACR) and the modified ACR (MACR) method, to evaluate 360° video quality. Furthermore, this part evaluates simulator sickness in participants viewing 360° video on an HMD and explores how their prior VR experience levels correlate with the occurrence of these symptoms. The third and final part of the thesis focuses on assessing viewing conditions and rating consistency. It involves conducting subjective quality assessment tests for 360° videos under different viewing conditions, such as standing and seated viewing, and providing a statistical analysis of the psychophysical and psychophysiological measures. This part also investigates the consistency of 360° video quality assessments through repeated subjective quality assessment tests under opportunity-limited conditions. It examines how quality assessments vary between the standing and seated viewing conditions and explores whether participants' subjective evaluations of 360° videos change over time or remain stable across repeated exposures.

Immersive media such as virtual reality, augmented reality, and 360° video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360° video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent for each user. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360° video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360° video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community. © 2024. M. Elwardy et al., licensed to EAI. This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited. doi:10.4108/eetinis.v11i1.4323. All Rights Reserved.

The recent advances in immersive technologies have been essential in the development of a wide range of novel standalone and networked immersive media applications. The concepts of virtual reality, augmented reality, and mixed reality relate to different compositions of real and computer-generated virtual objects. In this context, 360° video streaming has become increasingly popular offering improved immersive experiences when viewed on a head-mounted display (HMD). An important component in the development of novel immersive media systems are subjective tests in which participants assess the quality of experience of representative test stimuli. In this paper, the annotated ACR360 dataset is presented which is publicly available on GitHub. The ACR360 dataset contains a wide range of psychophysical and psychophysiological data that was collected in Subjective tests on 360° video quality. The test stimuli were shown on an HMD and rated according to the absolute category rating (ACR) and modified ACR (MACR) methods. To support an easy exploration and utilization of the ACR360 dataset by the research community, its structure on GitHub is described and a comprehensive illustration of analysis options are provided for each data category. The ACR360 dataset may be used for conducting meta-analysis in combination with other datasets to improve precision and to pursue research questions that cannot be answered by an individual study. © 2023 IEEE.

Subjective 360° video quality experiments provide the ground truth for developing related systems such that acceptable quality of experience is offered to the users. To obtain statistically meaningful insights from the data recorded in quality experiments, engaging a sufficiently large number of subjects is essential. In contrast to conventional videos, little has been reported on statistical measures that may guide the choice of the number of subjects required for 360° video quality experiments. In this paper, the standard deviation of opinion score (SOS) hypothesis, measuring the diversity among the opinion scores given by a panel of subjects, is used to reveal the number of subjects needed. It is shown that the number of subjects needed varies depending on a threshold on SOS characteristics in terms of diversity among opinion scores. © 2022 IEEE.

In this paper, we introduce the SSV360 dataset containing psy-chophysical and psychophysiological data gathered in a pilot study on the effect of standing and seated viewing of 360° videos on subjective quality assessment. The stimuli were shown to the participants on a head-mounted display for both viewing conditions. The ground truth contained in the SSV360 dataset may support, e.g., benchmarking of immersive media processing algorithms and developing of objective perceptual quality metrics. © 2022 IEEE.

Watching 360 degrees videos on head mounted displays (HMDs) allows viewers to explore scenes in all directions. In this paper, we focus on investigating the head movements of two participants for standing and seated viewing of a total of 720 360 degrees videos on HMDs. The head movements were recorded in a 360 degrees video quality assessment experiment which was repeated after a long and short break between sessions to study changes in viewing behavior over time. The analysis of the head movement data is provided as histograms of head rotations, head speed, head turns, and head trajectories. It is shown that the participants have their own distinct exploration behavior for standing viewing which becomes less different for seated viewing.

Due to the advances in head-mounted displays (HMDs), hardware and software technologies, and mobile connectivity, virtual reality (VR) applications such as viewing 360 degrees videos on HMDs have seen an increased interest in a wide range of consumer and vertical markets. Quality assessment of digital media systems and services related to immersive visual stimuli has been one of the challenging problems of multimedia signal processing. Specifically, subjective quality assessment of 360 degrees videos presented on HMDs is needed to obtain a ground truth on the visual quality as perceived by humans. Standardized test methodologies to assess the subjective quality of 360 degrees videos on HMDs are currently not as developed as for conventional videos and are subject to further study. In addition, subjective tests related to quality assessment of 360 degrees videos are commonly conducted with participants seated on a chair but neglect other options of consumption such as standing viewing. In this paper, we compare the effect that standing and seated viewing of 360 degrees videos on an HMD has on subjective quality assessment. A pilot study was conducted to obtain psychophysical and psychophysiological data that covers explicit and implicit responses of the participants to the shown 360 degrees video stimuli with different quality levels. The statistical analysis of the data gathered in the pilot study is reported in terms of average rating times, mean opinion scores, standard deviation of opinion scores, head movements, pupil diameter, galvanic skin response (GSR), and simulator sickness scores. The results indicate that the average rating times consumed for 360 degrees video quality assessment are similar for standing and seated viewing. Further, the participants showed higher resolving power among different 360 degrees video quality levels and were more confident about the given opinion scores for seated viewing. On the other hand, a larger scene exploration of 360 degrees videos was observed for standing viewing which appears to distract from the quality assessment task. A slightly higher pupil dilation was recorded for standing viewing which suggests a slightly more immersed experience compared to seated viewing. GSR data indicate a lower degree of emotional arousal in seated viewing which seems to allow the participants to better conduct the quality assessment task. Similarly, simulator sickness symptoms are kept significantly lower when seated. The pilot study also contributes to a holistic view of subjective quality assessment and provides indicative ground truth that can guide the design of large-scale subjective tests.

The development of immersive media systems and services relies on subjective tests that provide ground truth on the quality of experience as perceived by humans. In this paper, we investigate the opinion score consistency through a repeated subjective 360° video quality assessment experiment. The test stimuli are presented on a head-mounted display with participants given the task to rate the video quality in three repeated sessions for both standing and seated viewing. The statistical analysis of the data from the subjective tests aims at revealing if participants change their rating behavior over time or keep their opinion scores given to the test stimuli in standing and seated viewing consistent in a statistical sense. The experimental results are reported in terms of histograms of opinion scores, skewness and kurtosis of opinion scores, mean opinion scores, standard deviation of opinion scores (SOS), SOS fitting functions, and analysis of variance tests. The statistical analysis supports the conjecture that each participant has its own but consistent rating behavior throughout the respective sessions for standing and seated viewing. The finding that the quality rating behavior of the individual participant does not fundamentally change over time may, e.g., assist in scheduling subjective tests under pandemic conditions where experimental campaigns may need to be stalled for an unknown period of time. © 2021 IEEE.

Watching 360° videos on head-mounted displays (HMDs) has become more popular in recent years. As such, subjective quality assessment of 360° videos on HMDs is needed to obtain a ground truth on the quality as perceived by users. Due to the lack of standardized methods, the absolute category rating (ACR) with hidden reference (ACR-HR) method for conventional videos has also been used for subjective quality assessment of 360° videos on HMDs. A modified ACR (M-ACR) method tailored for assessing 360° videos on HMDs has recently been proposed. In this paper, we compare the ACR-HR and M-ACR with hidden reference (M-ACR-HR) method regarding subjective quality assessment of 360° videos on HMDs. The comparison is conducted subject to participants' experience with watching immersive media on HMDs. The findings include: 1) Average rating times to cast opinion scores is lower for the ACR-HR method irrespective of participants' experience, 2) Mean opinion scores are similar for both methods irrespective of participants' experience, 3) The M-ACR-HR method is more reliable, especially with experts and participants with no experience, 4) Discomfort is significantly lower for the ACR-HR method, especially, for participants with no experience. © 2020 IEEE.

Virtual reality (VR) has seen tremendous advances in head-mounted displays (HMDs), optics, media quality, and other improvements that facilitate immersive experiences. With the occurrence of new technologies like Cloud VR and networked VR video services, applications such as 360° video streaming are becoming more popular within the broader consumer markets. As a result, VR content is accessible to customers with rather different levels of experiences with immersive media, i.e., never, sometimes, or often use of VR. The question, therefore, arises to which degree simulator sickness is induced to viewers depending on their experiences with VR on HMDs. In this paper, simulator sickness is evaluated for 360° videos that were shown on an HTC Vive Pro HMD to participants having different levels of experience with VR on HMDs. The modified absolute category rating with hidden reference (M-ACR-HR) method was used in a subjective experiment for video quality assessment within two subsequent sessions along with a simulator sickness questionnaire (SSQ). A statistical analysis of the SSQ scores is performed to reveal the relationship between simulator sickness and participants' experiences with VR regarding: (1) Individual symptoms, (2) Pairwise comparison of symptoms, and (3) Symptom clusters of nausea, oculomotor, disorientation, and total score. It is shown that the simulator sickness symptoms, in general, are slightly or rarely perceived across the different experience levels for the selected 360° videos. The results indicate that the reported simulator sickness increases in the second session for participants that never used VR on HMDs. Sufficiently long breaks between sessions should therefore be accounted for in the M-ACR-HR method to avoid that simulator sickness influences quality rating. © 2020 IEEE.