The advances in 5G mobile networks are expected to enable immersive interconnected mobile multimedia systems. As humans are the final judges of the quality of immersive multimedia, it is essential to engage a suitable ground truth in the design of such systems. Databases annotated with results from subjective tests constitute such ground truth given as opinion scores, head movements, eye tracking data, psychophysiological data, and other data related to the viewers' behavior. On this basis, perception-based quality assessment of algorithms, systems, and services can be performed, and objective perceptual quality models can be developed. In this paper, a comprehensive survey of publicly available annotated 360-degree image and video databases is provided. The survey may guide the selection of ground truth on 360-degree images and videos to support quality assessment and modeling research. Further, the survey reveals the need for establishing new annotated databases that address the full range of subjective aspects of immersive multimedia. © 2019 IEEE.

The recent advances in immersive technologies have been essential in the development of a wide range of novel standalone and networked immersive media applications. The concepts of virtual reality, augmented reality, and mixed reality relate to different compositions of real and computer-generated virtual objects. In this context, 360° video streaming has become increasingly popular offering improved immersive experiences when viewed on a head-mounted display (HMD). An important component in the development of novel immersive media systems are subjective tests in which participants assess the quality of experience of representative test stimuli. In this paper, the annotated ACR360 dataset is presented which is publicly available on GitHub. The ACR360 dataset contains a wide range of psychophysical and psychophysiological data that was collected in Subjective tests on 360° video quality. The test stimuli were shown on an HMD and rated according to the absolute category rating (ACR) and modified ACR (MACR) methods. To support an easy exploration and utilization of the ACR360 dataset by the research community, its structure on GitHub is described and a comprehensive illustration of analysis options are provided for each data category. The ACR360 dataset may be used for conducting meta-analysis in combination with other datasets to improve precision and to pursue research questions that cannot be answered by an individual study. © 2023 IEEE.

In this paper, we introduce the SSV360 dataset containing psy-chophysical and psychophysiological data gathered in a pilot study on the effect of standing and seated viewing of 360° videos on subjective quality assessment. The stimuli were shown to the participants on a head-mounted display for both viewing conditions. The ground truth contained in the SSV360 dataset may support, e.g., benchmarking of immersive media processing algorithms and developing of objective perceptual quality metrics. © 2022 IEEE.

In this paper, we study the impact that the level of experience of participants with immersive multimedia has on 360° video quality assessment. An extensive subjective experiment using the modified absolute category rating with hidden reference (M-ACR-HR) method was conducted. The opinion scores obtained from the participants in this experiment along with other sensor data support this study. The statistical analysis is performed in terms of screening participants' distribution of opinion scores for consistency, average rating time, box plots related to mean opinion scores (MOSs) and differential MOS (DMOS). Detailed results for MOS and DMOS are also provided for reference and test sequences covering a wide range of qualities. Major findings include: (1) Screening detects many outliers for participants with no experience, (2) Average rating times show that experts learn the quality assessment task fast, (3) Participants with no experience provide lower opinion scores in average, (4) DMOS may alleviate the bias in quality rating caused by different levels of experience. © 2019 IEEE.

Virtual reality (VR) has seen tremendous advances in head-mounted displays (HMDs), optics, media quality, and other improvements that facilitate immersive experiences. With the occurrence of new technologies like Cloud VR and networked VR video services, applications such as 360° video streaming are becoming more popular within the broader consumer markets. As a result, VR content is accessible to customers with rather different levels of experiences with immersive media, i.e., never, sometimes, or often use of VR. The question, therefore, arises to which degree simulator sickness is induced to viewers depending on their experiences with VR on HMDs. In this paper, simulator sickness is evaluated for 360° videos that were shown on an HTC Vive Pro HMD to participants having different levels of experience with VR on HMDs. The modified absolute category rating with hidden reference (M-ACR-HR) method was used in a subjective experiment for video quality assessment within two subsequent sessions along with a simulator sickness questionnaire (SSQ). A statistical analysis of the SSQ scores is performed to reveal the relationship between simulator sickness and participants' experiences with VR regarding: (1) Individual symptoms, (2) Pairwise comparison of symptoms, and (3) Symptom clusters of nausea, oculomotor, disorientation, and total score. It is shown that the simulator sickness symptoms, in general, are slightly or rarely perceived across the different experience levels for the selected 360° videos. The results indicate that the reported simulator sickness increases in the second session for participants that never used VR on HMDs. Sufficiently long breaks between sessions should therefore be accounted for in the M-ACR-HR method to avoid that simulator sickness influences quality rating. © 2020 IEEE.

Watching 360° videos on head-mounted displays (HMDs) has become more popular in recent years. As such, subjective quality assessment of 360° videos on HMDs is needed to obtain a ground truth on the quality as perceived by users. Due to the lack of standardized methods, the absolute category rating (ACR) with hidden reference (ACR-HR) method for conventional videos has also been used for subjective quality assessment of 360° videos on HMDs. A modified ACR (M-ACR) method tailored for assessing 360° videos on HMDs has recently been proposed. In this paper, we compare the ACR-HR and M-ACR with hidden reference (M-ACR-HR) method regarding subjective quality assessment of 360° videos on HMDs. The comparison is conducted subject to participants' experience with watching immersive media on HMDs. The findings include: 1) Average rating times to cast opinion scores is lower for the ACR-HR method irrespective of participants' experience, 2) Mean opinion scores are similar for both methods irrespective of participants' experience, 3) The M-ACR-HR method is more reliable, especially with experts and participants with no experience, 4) Discomfort is significantly lower for the ACR-HR method, especially, for participants with no experience. © 2020 IEEE.

Due to the advances in head-mounted displays (HMDs), hardware and software technologies, and mobile connectivity, virtual reality (VR) applications such as viewing 360 degrees videos on HMDs have seen an increased interest in a wide range of consumer and vertical markets. Quality assessment of digital media systems and services related to immersive visual stimuli has been one of the challenging problems of multimedia signal processing. Specifically, subjective quality assessment of 360 degrees videos presented on HMDs is needed to obtain a ground truth on the visual quality as perceived by humans. Standardized test methodologies to assess the subjective quality of 360 degrees videos on HMDs are currently not as developed as for conventional videos and are subject to further study. In addition, subjective tests related to quality assessment of 360 degrees videos are commonly conducted with participants seated on a chair but neglect other options of consumption such as standing viewing. In this paper, we compare the effect that standing and seated viewing of 360 degrees videos on an HMD has on subjective quality assessment. A pilot study was conducted to obtain psychophysical and psychophysiological data that covers explicit and implicit responses of the participants to the shown 360 degrees video stimuli with different quality levels. The statistical analysis of the data gathered in the pilot study is reported in terms of average rating times, mean opinion scores, standard deviation of opinion scores, head movements, pupil diameter, galvanic skin response (GSR), and simulator sickness scores. The results indicate that the average rating times consumed for 360 degrees video quality assessment are similar for standing and seated viewing. Further, the participants showed higher resolving power among different 360 degrees video quality levels and were more confident about the given opinion scores for seated viewing. On the other hand, a larger scene exploration of 360 degrees videos was observed for standing viewing which appears to distract from the quality assessment task. A slightly higher pupil dilation was recorded for standing viewing which suggests a slightly more immersed experience compared to seated viewing. GSR data indicate a lower degree of emotional arousal in seated viewing which seems to allow the participants to better conduct the quality assessment task. Similarly, simulator sickness symptoms are kept significantly lower when seated. The pilot study also contributes to a holistic view of subjective quality assessment and provides indicative ground truth that can guide the design of large-scale subjective tests.

Immersive media such as virtual reality, augmented reality, and 360° video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360° video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent for each user. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360° video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360° video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community. © 2024. M. Elwardy et al., licensed to EAI. This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited. doi:10.4108/eetinis.v11i1.4323. All Rights Reserved.