Conference PaperPDF Available

Network Fortune Cookie: Using Network Measurements to Predict Video Streaming Performance and QoE

Authors:
Roberto Iraja Tavares da Costa Filho at Instituto Federal de Educação, Ciência e Tecnologia Sul-Rio-Grandense (IFSUL)
Roberto Iraja Tavares da Costa Filho
William Lautenschlager
William Lautenschlager
William Lautenschlager
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Nicolas Kagami at Universidade Federal do Rio Grande do Sul
Nicolas Kagami
Valter Roesler at Universidade Federal do Rio Grande do Sul
Valter Roesler
Show all 5 authorsHide
Download full-text PDF
Read full-text
Download citation

Figures

Figures - uploaded by Roberto Iraja Tavares da Costa Filho
Author content
All figure content in this area was uploaded by Roberto Iraja Tavares da Costa Filho
Content may be subject to copyright.
Content uploaded by Roberto Iraja Tavares da Costa Filho
Author content
All content in this area was uploaded by Roberto Iraja Tavares da Costa Filho on May 26, 2018
Content may be subject to copyright.
A preview of the PDF is not available
... Esses trabalhos avaliam o impacto de métricas de desempenho-como banda de rede, taxa de perdas de pacote e taxa de transmissão-no desempenho de aplicações específicas, incluindo vídeo [Casas et al. 2013, Gill et al. 2007, Shafiq et al. 2014, Chen et al. 2015, Ahmed et al. 2017. Entretanto, previsões, quando realizadas, são direcionadas a descoberta da opinião subjetiva dos usuários a respeito da qualidade de experiência (Mean Opinion Score, MOS) a partir das métricas de desempenho [da Costa Filho et al. 2016, Balachandran et al. 2013. Isso se deveà existência de diversos obstáculosà predição de engajamento. ...
... Tipicamente, os métodos existentes usam a taxa do vídeo ou estatísticas globais de eventos de stalls como previsores de QoE. Por exemplo, [da Costa Filho et al. 2016] estimam QoE através de umaárvore de decisão, tomando como variáveis independentes métricas de desempenho obtidas em um ambiente controlado. Essa abordagem não leva em conta transmissões adaptativas. ...
Previsão de Engajamento de Usuários Durante Transmissão Adaptativa de Vídeo ao Vivo
Conference Paper
Full-text available
  • May 2018
Os recentes esforços no desenvolvimento de algoritmos de adaptação e alocação de usuários têm contribuído de forma significativa para o aumento da qualidade de experiência (QoE) na distribuição de vídeo ao vivo na Internet. Entretanto, uma considerável parcela de sessões ainda sofre com baixa QoE, o que pode implicar em queda de engajamento dos usuários. Esse problema persiste uma vez que os provedores de serviço não conseguem prever a sáıda de um usuário e tomar medidas para evitá-la. Neste trabalho, propomos um modelo multi-estágio para predição de engajamento baseado em variáveis historicamente relacionadas a QoE. As sessões são agrupadas de acordo com suas similaridades de desempenho. Para cada grupo de sessões, modelos baseados emárvores de decisão são criados para prever (1) o tempo restante da sessão e (2) se o usuário permanecerá ou não no sistema pelos próximos n minutos. Experimentos com um conjunto de dados reais mostram uma significativa acurácia na predição de tempo restante e permanência, o que evidencia a viabilidade do uso das métricas de desempenho para prever engajamento de usuários.
View
... Many methods rely on Deep Packet Inspection (DPI), which is becoming infeasible due to the adoption of encryption protocols, on top of raising privacy issues [15], [16]. Other methods monitor the high-level network performance and use statistical models to map the objective metrics into indicators of user experience [17]. One problem commonly found on existing methods is how to cover the last mile of the end-to-end connection between server and client, since network monitoring tools usually require agents to be installed on the device to perform measurements. ...
View
... Um problema comum na Internet é o congestionamento, devido à redução de largura de banda ou presença de múltiplos usuários [Jiang, Sekar, and Zhang 2014], [Da Costa et al. 2016]. Para minimizá-lo, os mecanismos de controle de congestionamento para TCP são implementados [Cai, Shen, Pan, and Mark 2005]. ...
View
... Nela os usuários dão notas a aspectos relacionadosà qualidade do vídeo assistido. MOSé usada, por exemplo, por [da Costa Filho et al. 2016], que propõe um previsor para estimar o impacto de diversas métricas de qualidade de serviço (QoS) no valor de MOS. Apesar de confiável, a métrica MOS nãoé escalável, visto que a obtenção de um conjunto de referência representativo requer contribuição de uma grande parcela de usuários, o que nem sempreé trivial. ...
Um Modelo Sensível a Adaptação para Previsão de QoE na Transmissão de Vídeos ao Vivo
Conference Paper
Full-text available
  • Jun 2020
Today, video streaming is a well-established application on the internet. Providers are capable of transmitting video globally, in large-scale, although there are still some challenges, such as offering high quality of experience (QoE) for all clients. To face this problem, the content providers try to predict low QoE based on historical system performance to prevent the occurrence of new cases of low quality. In other words, the provider monitors the session performance (the average bitrate and rebuffering rate, for example) and applies methods to correlate this to a metric that expresses QoE, such as user engagement. This method allows the detection of the aspects that contribute to early session abandonment. However, the success of this approach depends on the accuracy of the QoE prediction: past works report an accuracy up to 70% using the pre-mentioned correlation between performance and engagement. In this work, we propose a new method for engagement prediction through the use of the client adaptation regime in replacement of the pre-mentioned performance metrics. We show that this new approach is capable of achieving 81% of prediction accuracy. We also present a case study for the CDN control plan that uses QoE prediction. We found that its use can lead to potential gains in user engagement in comparison to the default provider approach. Resumo. As transmissões de vídeo ao vivo já romperam a fronteira da escala global e agora enfrentam um novo desafio: oferecer alta qualidade de experiência (QoE) de maneira uniforme. Para atacar esse problema, diversas abordagens utilizam informações históricas relacionadas ao desempenho de transmissão para inferir a qualidade de experiência (QoE) de suas sessões. Prever a QoE de maneira acurada permite que provedores de serviço, proativamente, estimem e aloquem recursos a cada um de seus clientes. No entanto, o sucesso dessa alocação de recursos customizada depende de uma previsão acurada de QoE: tipicamente, a previsãoé feita por meio de métricas de desempenho como taxa de interrupções e bitrate médio e alcançam uma acurácia abaixo de 70%. Neste trabalho, apresentamos uma nova abordagem, que correlaciona a QoE do cliente com o perfil de adaptação da sua mídia. Nós mostramos que o novo conjunto de métricas proposto aumenta a acurácia de previsão para 81%. Também apresentamos um estudo de caso para alocação de clientes usando o previsor e demonstramos que existe um potencial de duplicação da QoE geral se comparadoà alocação padrão empregada pelo provedor.
View
Estimating Video on Demand QoE from Network QoS through ICMP Probes
Article
  • Nov 2021
With the increasing traffic of vod, network providers are seeking to deliver high qoe for their users. Many methods have been proposed to assess vod-related qoe. Some of them rely on client instrumentation and reporting qoe information to network elements, such as Server and Network Assisted DASH, others are based on statistical methods that make qoe inferences using monitored network conditions, such as throughput and delays. In this article, we present a practical method to estimate qoe for vod using the widely supported icmp probes. Measured network conditions are used as input to a ml model that estimates qoe in terms of mos, based on the ITU-T P.1203 Recommendation. The estimation encompasses video quality switches and playback stalls. We estimate mos with an average rmse of 1.05 for a catalog of 25 different videos, training a model with sessions of the shortest video, and evaluating the generalization to the full catalog. We performed experiments using a virtualized setup as well as in a Wide Area Network.
View
View
A Hybrid Control Scheme for 360-Degree Dynamic Adaptive Video Streaming Over Mobile Devices
Article
  • Feb 2021
A 360-degree streaming system can provide immersive, interactive, and autonomous experiences surrounding the user by viewpoint changing to see different angles of the 360-degree video. Due to the limited and highly dynamic cellular network conditions, high-resolution 360-degree video playback over mobile devices often suffer from playback freezing, and inevitable bandwidth waste appears in delivering those out-of-viewpoint parts. In this paper, a hybrid control scheme is presented for segment-level continuous bitrate selection and tile-level bitrate allocation for 360-degree streaming over mobile devices to increase users' quality of experience. First, a deep reinforcement learning (RL) method is proposed to predict the segment bitrate and avoid playback freezing events. Second, a viewpoint-prediction-map-based cooperative bargaining game theory is proposed for bitrate allocation optimization to choose a suitable bitrate for each tile to reduce unreasonable bandwidth waste. The proposed scheme is compared with state-of-the-art approaches over a wide variety of mobile network conditions with multiple viewpoint traces and 360-degree video contents. Experimental results indicate that the proposed method outperforms the state-of-the-art approaches in terms of different experimental objectives over mobile devices.
View
View
View
View
UAMA: A unified architecture for active measurements in IP networks - End-to-end objetive quality indicators
Conference Paper
Full-text available
  • May 2007
The necessity to keep acceptable service quality levels in IP networks is undeniable nowadays, particularly with the improvement of the next generation networks. This promotes the use and justifies the need for solutions that enable network measurement and monitoring. Network measurement and performance data acquisition is currently a target of many research and development efforts, mainly due to its applicability in different situations, including network traffic engineering. In this field, there are two measurement approaches: passive and active. Passive measurement is based on the observation of ordinary traffic of an operational network, without the introduction of any kind of traffic in the monitoring probe. On the other hand, the active approach tries to acquire performance indicators using a controlled traffic generation and transmission through the analyzed network. Knowing the potentialities and deficiencies of such techniques, and specifically directing our interest to IP network traffic engineering, this paper emphasizes the active measurement particularities, proposing the project, modeling and implementation of a standardized and extensible architecture for this kind of measurement. We try to join together different metric acquisition processes in a unified system. To do so, we present the architecture entities and the conceived and implemented protocol that are used for communication between these entities, highlighting the technique used to minimize the traffic impact in the ongoing network operation. Finally, we also introduce the idea behind the architecture flexibility through its plug-in approximation.
View
Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization
Article
Full-text available
  • Oct 2015
  • INT J ARTIF INTELL T
In a typical supervised data analysis task, one needs to perform the following two tasks: (a) select an optimal combination of learning methods (e.g., for variable selection and classifier) and tune their hyper-parameters (e.g., K in K-NN), also called model selection, and (b) provide an estimate of the performance of the final, reported model. Combining the two tasks is not trivial because when one selects the set of hyper-parameters that seem to provide the best estimated performance, this estimation is optimistic (biased/overfitted) due to performing multiple statistical comparisons. In this paper, we discuss the theoretical properties of performance estimation when model selection is present and we confirm that the simple Cross-Validation with model selection is indeed optimistic (overestimates performance) in small sample scenarios and should be avoided. We present in detail and investigate the theoretical properties of the Nested Cross Validation and a method by Tibshirani and Tibshirani for removing the estimation bias. In computational experiments with real datasets both protocols provide conservative estimation of performance and should be preferred. These statements hold true even if feature selection is performed as preprocessing.
View
View
A Survey on Quality of Experience of HTTP Adaptive Streaming
Technical Report
Full-text available
  • Apr 2014
Changing network conditions pose severe problems to video streaming in the Internet. HTTP adaptive streaming (HAS) is a technology employed by numerous video services that relieves these issues by adapting the video to the current network conditions. It enables service providers to improve resource utilization and Quality of Experience (QoE) by incorporating information from different layers in order to deliver and adapt a video in its best possible quality. Thereby, it allows taking into account end user device capabilities, available video quality levels, current network conditions, and current server load. For end users, the major benefits of HAS compared to classical HTTP video streaming are reduced interruptions of the video playback and higher bandwidth utilization, which both generally result in a higher QoE. Adaptation is possible by changing the frame rate, resolution, or quantization of the video, which can be done with various adaptation strategies and related client- and server-side actions. The technical development of HAS, existing open standardized solutions, but also proprietary solutions are reviewed in this paper as fundamental to derive the QoE influence factors that emerge as a result of adaptation. The main contribution is a comprehensive survey of QoE related works from human computer interaction and networking domains, which are structured according to the QoE impact of video adaptation. To be more precise, subjective studies that cover QoE aspects of adaptation dimensions and strategies are revisited. As a result, QoE influence factors of HAS and corresponding QoE models are identified, but also open issues and conflicting results are discussed. Furthermore, technical influence factors, which are often ignored in the context of HAS, affect perceptual QoE influence factors and are consequently analyzed. This survey gives the reader an overview of the current state of the art and recent developments. At the same time, it targets networking res- archers who develop new solutions for HTTP video streaming or assess video streaming from a user centric point of view. Therefore, this paper is a major step toward truly improving HAS.
View
Performance-Estimation Properties of Cross-Validation- Based Protocols with Simultaneous Hyper-Parameter Optimization
Conference Paper
Full-text available
  • May 2014
  • INT J ARTIF INTELL T
In a typical supervised data analysis task, one needs to perform the following two tasks: (a) select the best combination of learning methods (e.g., for variable selection and classifier) and tune their hyper-parameters (e.g., K in K-NN), also called model selection, and (b) provide an estimate of the perfor-mance of the final, reported model. Combining the two tasks is not trivial be-cause when one selects the set of hyper-parameters that seem to provide the best estimated performance, this estimation is optimistic (biased / overfitted) due to performing multiple statistical comparisons. In this paper, we confirm that the simple Cross-Validation with model selection is indeed optimistic (overesti-mates) in small sample scenarios. In comparison the Nested Cross Validation and the method by Tibshirani and Tibshirani provide conservative estimations, with the later protocol being more computationally efficient. The role of strati-fication of samples is examined and it is shown that stratification is beneficial.
View
On User-Centric Tools for QoE-Based Recommendation and Real-Time Analysis of Large-Scale Markets
Article
Full-text available
  • Sep 2014
This article focuses on mechanisms that empower users with quality of experience recommendations and smart real-time analytics. It presents a user-centric recommendation system (called u-map) that enables users to collect network measurements and subjective opinion scores about the performance of various services. It also cross-correlates measurements obtained by u-map to provide geo-statistics, user profiles, and quality of experience prediction models for different services. The article also presents CoRLAB, a modular multi-layer framework for modeling and assessing various markets, services, and their evolution under a diverse set of customer populations and conditions. U-map feeds CoRLAB with user measurements and feedback in (semi) real-time. The article discusses how u-map and CoRLAB have been used to analyze telecommunication markets and services. It highlights the main research results, challenges, and potential research directions.
View
Developing a predictive model of quality of experience for internet video
Conference Paper
  • Aug 2013
  • COMPUT COMMUN REV
Improving users' quality of experience (QoE) is crucial for sustaining the advertisement and subscription based revenue models that enable the growth of Internet video. Despite the rich literature on video and QoE measurement, our understanding of Internet video QoE is limited because of the shift from traditional methods of measuring video quality (e.g., Peak Signal-to-Noise Ratio) and user experience (e.g., opinion scores). These have been replaced by new quality metrics (e.g., rate of buffering, bitrate) and new engagement centric measures of user experience (e.g., viewing time and number of visits). The goal of this paper is to develop a predictive model of Internet video QoE. To this end, we identify two key requirements for the QoE model: (1) it has to be tied in to observable user engagement and (2) it should be actionable to guide practical system design decisions. Achieving this goal is challenging because the quality metrics are interdependent, they have complex and counter-intuitive relationships to engagement measures, and there are many external factors that confound the relationship between quality and engagement (e.g., type of video, user connectivity). To address these challenges, we present a data-driven approach to model the metric interdependencies and their complex relationships to engagement, and propose a systematic framework to identify and account for the confounding factors. We show that a delivery infrastructure that uses our proposed model to choose CDN and bitrates can achieve more than 20\% improvement in overall user engagement compared to strawman approaches.
View
View
View
‘’Is QoE estimation based on QoS parameters sufficient for video quality assessment?”
Conference Paper
  • Jun 2013
Internet Service providers offer today a variety of of audio, video and data services. Traditional approaches for quality assessment of video services were based on Quality of Service (QoS) measurement. These measurements are considered as performance measurement at the network level. However, in order to make accurate quality assessment, the video must be assessed subjectively by the user. However, QoS parameters are easier to be obtained than the QoE subjective scores. Therefore, some recent works have investigated objective approaches to estimate QoE scores based on measured QoS parameters. The main purpose is the control of QoE based on QoS measurements. This paper presents several solutions and models presented in the literature. We discuss some other factors that must be considered in the mapping process between QoS and QoE. The impact of these factors on perceived QoE is verified through subjective tests.
View