We focus on the identification of the key influence factors impacting QoE, taking into account user, system, and context-related factors. Research is conducted using quality assessment techniques involving the collection of both subjective and objective metrics. Of particular interest are emerging immersive services, such as AR/VR communications, 360-degree video streaming, networked games, and multiparty audiovisual communication services. Understanding and modeling the factors that impact QoE inherently calls for a multidisciplinary approach, combining knowledge from engineering, psychology, and user interface design. The modeling process involves using both analytical approaches and “black box” approaches based on various machine learning techniques.

Modeling QoE for cloud gaming

Cloud gaming has been recognized as a promising shift in the online game industry, with the aim being to deliver high-quality graphics games to any type of end-user device. The concepts of cloud computing are leveraged to render the game scene as a video stream which is then delivered to players in real-time. Given high bandwidth and strict latency requirements, a key challenge faced by cloud game providers lies in configuring the video encoding parameters so as to maximize player QoE while meeting bandwidth availability constraints. On the other hand, network operators are interested in understanding the relationships between parameters measurable in the network and cloud gaming QoE, to be able to invoke QoE-aware network management mechanisms.

We conduct subjective studies aimed at identifying QoE-driven video encoding adaptation strategies. Empirical results are used to derive analytical QoE estimation models as functions of bitrate and framerate, while also taking into account game type and player skill. Given that results indicate that different QoE-driven video adaptation policies should likely be applied for different types of games, we further study objective video metrics that may be used to classify games for the purpose of choosing an appropriate and QoE-driven video codec configuration strategy.

Based on the analyzed objective video and gameplay characteristics, it is possible to categorize cloud-based games using cluster analysis. E.g., we used k-means clustering to group games into 3 clusters characterized by objective video metrics (IBS, PFIM) and intensity of user actions (APM). However, additional cluster analysis techniques could be applied and its results be compared to the obtained results. 

In an effort to contribute to ongoing research efforts addressing challenges related to characterization of cloud gaming video streams, in terms of video coding parameters, spatial and temporal parameters, different game genres, traffic characteristics, and player input rate, we also perform data collection of gameplay recordings, application logs, and network traces corresponding to various game genres. Obtained data in this process is highly relevant for future work on automating the process of configuring video encoding settings and estimating video quality for cloud gaming.

5Gaming

This relatively new approach for gaming content delivery has since recently attracted key market players such as Sony, Microsoft, Google, and Amazon. Cloud gaming significantly increases the requirements on the network in terms of both bandwidth usage and latency. Deployment of 5G networks, combined with cloud gaming, will allow for ubiquitous gaming - playing any game, on any platform, anywhere. For mobile network operators, cloud gaming will be one of the most important use cases for attracting customers interested in the entertainment capabilities of the new generation of mobile networks. In this domain, we are cooperating with the Croatian telecom in the scope of the project 5Gaming. The goal of the project is to set up a demo environment and, through a number of scenarios related to gaming, demonstrate the advantages of the 5G network. The scenarios include networked multiplayer virtual reality (VR) gaming, mobile cloud gaming, and VR cloud gaming. 

Selected publications:

• I. Slivar, K. Bačić, I. Oršolić, L. Skorin-Kapov, M. Sužnjević, "CGD: A Cloud Gaming Dataset with Gameplay Video and Network Recordings", in the 13th ACM Multimedia Systems Conference (MMSys 2022), Athlone, Ireland, June 2022.

• I. Slivar, L. Skorin-Kapov, M. Sužnjević, "QoE-Aware Resource Allocation for Multiple Cloud Gaming Users Sharing a Bottleneck Link", 3rd International Workshop on Quality of Experience Management (QoE-Management 2019), in conjunction with ICIN 2019, Paris, France, 2019.

• I. Slivar, M. Sužnjević, L. Skorin-Kapov, "Game Categorization for Deriving QoE-Driven Video Encoding Configuration Strategies for Cloud Gaming", ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), vol. 14, issue 3, June 2018.
• I. Slivar, L. Skorin-Kapov, M. Sužnjević, "Cloud Gaming QoE Models for Deriving Video Coding Adaptation Strategies", Proc. of ACM Multimedia Systems (MMSys'16), Klagenfurt, Austria, 2016. 
• M. Sužnjević, I. Slivar, L. Skorin-Kapov, "Analysis and QoE Evaluation of Cloud Gaming Service Adaptation Under Different Network Conditions: the Case of NVIDIA GeForce NOW", in proc. of 8th International Conf. on Quality of Multimedia Experience (QoMEX 2016), Lisbon, Portugal, 2016

Immersive VR and AR services

Advances in the field of VR have focused on improvements in VR interaction techniques, including support for different locomotion, or navigation, methods. We study how different navigation methods influence the user perception of immersion and the occurrence of simulator (or VR) sickness. Going beyond subjective assessment methods, we collect physiological measures (e.g., EEG, EDA) to explore correlations with simulator sickness. Further focusing on VR gaming, we conduct subjective studies investigating the gaming experience, as well as its aftereffects (e.g., simulator sickness, discomfort, fatigue, reaction time), with respect to different human, context, and system factors. 

In regard to multiuser networked immersive environments, we explore the impact of various system and network QoS factors on user QoE, in scenarios involving AR- and VR-based games and collaborative virtual environments. In cooperation with the company DivIT, we investigate the impact of different compression techniques on volumetric video quality, and explore their suitability for pre-recorded content, as well as content streamed in real-time. In the scope of the Q-MERSIVE project, we focus on both AR and VR media in light of recent technological advancements, especially with respect to the capabilities of 5G networks.

Selected publications:

• S. Vlahovic, M. Suznjevic, L. Skorin-Kapov, "A Framework for the Classification and Evaluation of Game Mechanics for Virtual Reality Games". Electronics 2022, 11, 2946. https://doi.org/10.3390/electronics11182946
• I. Slivar, S. Vlahović, M. Šilić, L. Skorin-Kapov, M. Sužnjević, "The Impact of Network and Social Context on Quality of Experience for Competitive Multiplayer Virtual Reality Games", in 2nd Workshop on Game Systems (GameSys'22), Athlone, Ireland, June 2022.
• S. Vlahović, M. Sužnjević, L. Skorin-Kapov. "A survey of challenges and methods for Quality of Experience assessment of interactive VR applications". Journal on Multimodal User Interfaces (2022). https://doi.org/10.1007/s12193-022-00388-0
• S. Vlahović, M. Sužnjević, N. Pavlin-Bernardić, L. Skorin-Kapov, "The Effect of Different VR Game Interactions on Discomfort, Cybersickness, and Reaction Time", to appear in 13th International Conference on Quality of Multimedia Experience (QoMEX 2021), Montreal, Canada, June 2021.
• S. Vlahović, M. Manđurov, M. Sužnjević, and L. Skorin-Kapov. "Usability Assessment of a Wearable Video-Communication System", 12th International Conference on Quality of Multimedia Experience (QoMEX 2020), Athlone, Ireland, May 2020.
• S. Vlahović, M. Sužnjević, and L. Skorin-Kapov. "Challenges in Assessing Network Latency Impact on QoE and In-Game Performance in VR First Person Shooter Games.", 15th International Conference on Telecommunications (ConTEL), Graz, Austria, July 2019
• S. Vlahović, M. Sužnjević, and L. Skorin-Kapov. "The Impact of Network Latency on Gaming QoE for an FPS VR Game.", 11th International Conference on Quality of Multimedia Experience (QoMEX 2019), Berlin, Germany, June 2019
• S. Vlahović, M. Sužnjević, and L. Skorin-Kapov. "Subjective Assessment of Different Locomotion Techniques in Virtual Reality Environments." 10th International Conference on Quality of Multimedia Experience (QoMEX 2018), Sardinia, Italy, May 2018.

Modeling and assessment of QoE for Remote Reality

We define Remote Reality as a technology that, through the appropriate network infrastructure, enables an immersive experience of a remote real environment in real-time with latency low enough to enable an interactive experience in very dynamic scenarios. One of these scenarios is First-person view (FPV) drone piloting, a method of controlling a radio-controlled (RC) vehicle from the driver's or pilot's viewpoint. Because of the specific characteristics corresponding to FPV, such as low latency and high bandwidth requirements, there is a need to develop specific Quality of Experience (QoE) models. Due to strict latency requirements, FPV technology stagnated, with analog video transmission currently representing the most common implementation. The other scenario is remote control of different robots from a human operator or from an Artificial Intelligence (AI). In the case of human operator we are considering the QoE of the operator, considering different network conditions, while in the case of AI we are considering Quality of Service (QoS)m as performance of the robot can only be objectively evaluated.

We are collaborating with the company Orqa on the DRUNE project. Our goal is to develop an FPV system based exclusively on digital technology. As a part of our research, we conduct user studies to assess QoE in virtual and real drone flight scenarios.

In the case of remote robot control, we are collaborating with a swedish company Byte Motion. The goal of the coopeartion is to evaluate different options for remote robot control via the network and using simulation and implementation of the different aspect of the robot control system in Unity game engine.

Selected publications:

• M. Šilić, M. Sužnjević, and L. Skorin-Kapov. "QoE Assessment of FPV Drone Control in a Cloud Gaming Based Simulation", to appear in 13th International Conference on Quality of Multimedia Experience (QoMEX 2021), Montreal, Canada, June 2021.

WebRTC-based mobile multiparty audiovisual telemeetings

Managing interactive and multiparty video conferencing services requires an understanding of the key underlying QoE influence factors. A key challenge faced by service providers lies in configuring the video encoding parameters so as to maximize participant QoE while meeting resource (network and mobile device) availability constraints. Currently developed QoE models can for the most part be applied to two interlocutors and in desktop environments. However, there is a lack of studies that focus on modelling and optimizing QoE for such services in a mobile context, and in particular in the case of multiparty scenarios. Multiparty video call optimization is thus a challenging task due to dynamic wireless networks, heterogeneous mobile devices, and contexts. Hence, there is a need to investigate thresholds within video encoding parameters that can be used to determine optimal adaptation strategies for mobile multiparty video conferencing services. We study how to specify QoE-driven service adaptation for mobile multiparty video conferencing calls under variable resource availability. Extensive subjective user tests are conducted to identify key QoE influence factors and provide input for deriving QoE models.

Selected publications:

• D. Vučić, L. Skorin-Kapov, "QoE Assessment of Mobile Multiparty Audiovisual Telemeetings", IEEE Access, Vol. 8, 2020, DOI: 10.1109/ACCESS.2020.3000467
• D. Vučić, L. Skorin-Kapov, "The Impact of Packet Loss and Google Congestion Control on QoE for WebRTC-based Mobile Multiparty Audiovisual Telemeetings", in Proc. of the 25th International Conference on MultiMedia Modeling (MMM 2019), Thessaloniki, Greece, Jan. 2019
• D. Vučić, L. Skorin-Kapov, M. Sužnjević, "The impact of bandwidth limitations and video resolution size on QoE for WebRTC-based mobile multi-party video conferencing", in Proc. of the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems, PQS, Aug. 2016, Berlin
• D. Vučić, L. Skorin-Kapov. "The Impact of Mobile Device Factors on QoE for Multi-Party Video Conferencing via WebRTC", Proc. of the 13th International Conference on Telecommunications, ConTEL, Graz, Austria, July 2015. 

Multidimensional analysis of Web browsing QoE

The increasing use of mobile devices for Web browsing has driven a rising interest in understanding and enhancing the user experience when accessing mobile Web sites. We study multiple factors that impact the quality of the user experience while browsing the mobile Web, focusing on design, information quality, and loading time. Based on the results of subjective studies, we model QoE in the context of browsing information, thematic, and e-mail portals via browsers on smartphones and tablet devices. Results have shown the existence of significant effects of examined factors on QoE, while mutual relations between QoE and multiple perceivable characteristics of the experience that contribute to its quality, referred to as QoE features, are modelled, identifying the importance of the distinct dimensions in terms of overall user perceived QoE.

Selected publications:

• S. Baraković, L. Skorin-Kapov, "Survey of Research on Quality of Experience Modelling for Web Browsing", in Quality and User Experience, Springer, available online July 2017.
• S. Baraković, L. Skorin-Kapov, "Modeling the relationship between design/performance factors and perceptual features contributing to Quality of Experience for mobile Web browsing", Computers in Human Behavior, Vol 74, Sept 2017 (available online April 2017)
• S. Baraković, L. Skorin-Kapov, "Multidimensional Modelling of Quality of Experience for Mobile Web Browsing", Computers in Human Behavior, Sept. 2015, Vol. 50
• M. Varela, L. Skorin-Kapov, T. Mäki and T. Hoßfeld. "QoE in the Web: A Dance of Design and Performance", in Proceedings of the 7th International Workshop on Quality of Multimedia Experience (QoMEX 2015), Costa Navarino, Greece, May 2015.

Analysis of Player Behavior and QoE in mobile games

Gaming on mobile devices is a norm these days. More than 50% of the market revenue is created on mobile platforms. The most prominent business model is Free To Play (F2P). In F2P games the game itself is free, but there are different monetization options such as purchasing virtual items, or different in game boosts. Studying behavior of players in such games can yield better QoE of the players as well as better revenue for the game developers and publishers. Information such as  session duration, frequency of play, gameplay context data can say a lot about how players are playing the game and how much are they enjoying it.

In this area we have been collaborating with the company Nanobit.  As a part of our research, we conduct several data gathering sessions as well as deep analysis of the gathered data to provide insights regarding player behavior and QoE.

Selected publications:

• K. Mustač, K. Bačić, L. Skorin-Kapov, and M. Sužnjević: "Predicting Player Churn of a Free-to-Play Mobile Video Game Using Supervised Machine Learning". Applied Sciences, 12(6), 2795 (2022).