Towards a Crowdsourced Network Measurements Analyzer (CNMA) for the Streaming Service

Abstract

This exploratory work introduces a new data an-alyzer system, called CNMA (Crowdsourced Network Measurements Analyzer). CNMA aims at addressing the root cause of the bad performance of typical internet applications such as video on demand streaming and web search considering the 3G/4G radio access network. In particular, we study the related physical radio access parameters to the timeout status of transport connections and application throughput using the "5Gmark" crowdsourcing full tests. Thus, we consider dataset of LTE technology collected via measurements from one of the major French mobile operators in Ile-de-France region. To perform the analysis, CNMA is composed of three different modules. The first one is the storage module for storing the Crowdsourced Network Measurements (CNMs) raw data of "5Gmark" tool. The second module is the batch unit. The last module consists on the visualization (dashboard) one for plotting and reporting the analysis results. The main expected outcome of CNMA is the impact of the bit rate combined with other parameters of the physical layer such as the Received Signal Strength and the Reference Signal Received Quality (RSRQ) on the application throughput and their correlation to the connection timeout phenomena.

Full text

Towards a Crowdsourced Network Measurements
Analyzer (CNMA) for the Streaming Service
Lamine Amour
ESME Sudria - Paris - France
lamine.amour@esme.fr
Abdulhalim Dandoush
ESME Sudria - Paris - France
dandoush@esme.fr
Abstract—This exploratory work introduces a new data an-
alyzer system, called CNMA (Crowdsourced Network Measure-
ments Analyzer). CNMA aims at addressing the root cause of the
bad performance of typical internet applications such as video on
demand streaming and web search considering the 3G/4G radio
access network. In particular, we study the related physical radio
access parameters to the timeout status of transport connections
and application throughput using the ”5Gmark” crowdsourcing
full tests. Thus, we consider dataset of LTE technology collected
via measurements from one of the major French mobile operators
in Ile-de-France region. To perform the analysis, CNMA is
composed of three different modules. The first one is the storage
module for storing the Crowdsourced Network Measurements
(CNMs) raw data of ”5Gmark” tool. The second module is
the batch unit. The last module consists on the visualization
(dashboard) one for plotting and reporting the analysis results.
The main expected outcome of CNMA is the impact of the
bit rate combined with other parameters of the physical layer
such as the Received Signal Strength and the Reference Signal
Received Quality (RSRQ) on the application throughput and
their correlation to the connection timeout phenomena.
Index Terms—Crowdsourced Network Measurements, Crowd-
sourcing, Data collection, LTE Mobile Network Video Streaming,
5Gmark tool.
I. INTRODUCTION
With the Internet democratization use, monitoring the Qual-
ity of Experience (QoE) nowadays is crucial for both the
Internet Service Providers (ISPs) and end-users customers,
where most ISPs try to retain their customers by monitoring
whether or not they are satisfied with the services offered. In
fact, with the absence of the ISP’s dataset measured at the base
stations side due to privacy and economic policies issues, the
measurement techniques at the end-user side for assessing the
QoE has attracted the attentions of many research works. The
goal often is to understand root causes of bad performance
for some services/applications sensitive to a particular metrics
such as certain delay or a given packet loss level. Users can
face to common undesirable phenomena like slow playback,
playback error, connection timed-out to server, and video
buffering [1].
According to [2], Crowdsourced Network Measurements
(CNMs) provide insights beyond the network layer and offer
performance and other measurements at the application and
user-level towards real user perceived quality (QoE) and
service/application issues. Consequently, we had built our own
CNMs dataset and developed our own analyzer in order to
assess and to evaluate the impact of some physical environ-
mental parameters on the QoE perceived by mobile end-users
of a popular service, i.e., Youtube on-demand video streaming.
Therefore, this ongoing work presents a new data analyzer
system, called CNMA (Crowdsourced Network Measurements
Analyzer). This system is composed of three different modules
named storage, batch and visualization (dashboard) modules.
The remainder of the article is organized as follows: in
Section II, we introduce overall description of the CNMA
analyzer system. In Section III, we present the used tool
in this work and the collected CNMs dataset. In Section
IV, the implementation of CNMA scenario that addresses the
YouTube video streaming is described. The correlation of the
bit rate video (bitrate) and particular physical parameters, like
LT ERSRQ and LT Estreng th and their impact on the QoE
are analyzed. Finally, the conclusion and future works are
presented in the last section.
II. CROWDSOURCED NETWORK MEASUREMENTS
ANALYZER (CNMA)
In this section, we overview our approach for the im-
plementation of our data analyzer system, called CNMA
(Crowdsourced Network Measurements Analyzer). Its overall
architecture is presented in Figure (1).
A main objective of the CNMA consists on providing a
Python-based application that uses primary data (raw data)
of 5Gmark tool, in order to analyze and to plot the impact of
physical parameters and bit rate video on the YouTube High-
definition (HD) video Streaming. It will provide an automated
CNMs analysis with a good understanding, in real time, of the
crowdsourcing datasets. For this, it uses the following three
modules.
•Storage module :
In the first module, the raw data downloaded from ”5-
Gmark” portal is saved. It serves as a base to store
this raw data that we’ll use in the second module. The
used format here is Comma-separated values (CSV), but
this module can be extended according to the future
expectations to include others formats like text (.txt).
•Batch module :
In the second module, the stored data in the first module
is used to prepare a clean data that we analyze here.
This module is built using free machine learning library978-1-6654-4005-9/21/$31.00 ©2021 IEEE

Fig. 1. Overall architecture of Crowdsourced Network Measurements Analyzer (CNMA)
”scikit-learn” from the Python programming language.
This library features various different processing opera-
tions, like encoding and normalization, in addition to clas-
sification and regression algorithms including decision
trees that we implement here. We choose to use decision
trees because of their various advantages as (i) a great
utility for data analysis and machine learning because
they break complex data down into more manageable
parts and (ii) a simple presentation form (graphical) that
shows a clear path to a decision. Note that 75% of the
data is used for training and the rest for testing.
•Visualization module :
The third module of our proposal consists on the vi-
sualization module in a dashboard interface format. It
allows users to analyze and to understand the achieved
analysis in the previous module by plotting three kinds
of results. The first one is a matrix correlation (Pearson
correlation) that presents the dependencies or associations
between all the selected parameters in the second module.
The second kind of result consists on visualizing the
relation between three parameters by representing them
on three dimensions (3D) plot. The third kind of result
is presenting the decision tree in graphical format using
the ”pydotplus” library from Python.
Indeed, we indicate CNMA proposal is not just for plotting
and analyzing the impact of the selected parameters on the
performance of video streaming service, but also a complete
system that presents the output in a simple and easy inter-
pretive way (i.e., decision trees format, correlation matrix,
etc.), that facilitates the interpretation without having special
mathematical knowledge. Finally, we notice that our analyzer
is made public in the Gitlab repository [4], and we indicate
that this solution can be enhanced, for example, by taking into
account others data format in the first module, adding cross
validation and other ML methods in the second module.
III. CROWDSOURCED NETWORK MEASUREMENTS
A. Measurement tool
The Crowdsourced Network Measurements (CNMs) collec-
tion device might be the user terminal (smartphone, tablet, etc.)
or a special device associated with the user terminal [3]. In this
study, the 5Gmark tool is used like the study [2]. It consists on
user terminal collection approach that is Android smartphone.
It allows to measure the cellular connection of your mobile
through 3modes: ”Speed Test”, ”Full Test” and ”Drive test”.
In practice, the ”Full Test” integrates in addition to the ”Speed
Test” flow values the measurement of real customer usage
(YouTube streaming and web browsing). The ”Drive Test”
represents a test cycle, set of ”Speed Tests” or ”Full Tests”,
which runs automatically with a tests number counter (5,10,
20, etc.) and an interval in each test in minutes (by default
0). The choice of this tool is justified by several reasons, of
which we can quote [2], [3]:
•Active data collection: Using active approach, to collect
measurements, lets to configure/emulate behaviour of
different services (web browsing, video streaming, file
downloading, etc.).
•Variety of services: It consider several types of tests
for popular applications such as latency test (latency),

download data test (download), upstream data transfer
(upload), web browsing (web), video viewing in stream-
ing (video);
•Simplicity: In addition to allowing free access to data via
the personal portal, it also gives total visibility on the
quality of the connection environment.
•Detailed test report: It presents information that concerns
success or failure of the service being tested. In addition,
it also gives additional indication of the type of problem.
Typically for the video service, it indicates whether it
is video timeout (T I ME OU T ) or video service failed
(F AI LLED).
B. Dataset and parameters description
We present in this part our collected measurements and
the selected parameters in this study which have identified
as physical and application parameters. A part of our dataset
is made free. A detailed description and sources can be found
in the Gitlab repository [5].
This dataset is composed by trace measurements from the
5Gmark tool. The initial dataset contains more than 60000
customer-side cellular measurements collected from one of the
major French mobile operators, through two mobility models
(bus and train). More than 50 Go of data was downloaded
for five different applications (latency, download, upload, web
browsing and video) and for 3different technologies (LTE
(4G), HSPAP (HSPA+/3G++), HSPA (HSPA+/3G+)). The test
campaign took place during the month of March 2021 in the
Ile-de-France region in France.
In this exploratory work, we are focusing on LTE (4G)
technology (like [6]), which represents more than 80.5% of
the measurements collected. Table (2) reports basic statistics
concerning all the available services got from the considered
tool with the LT E mobile technology.
Fig. 2. Statistics concerning latency, download, upload, web browsing and
video services using LTE technology
From the Figure (2), we observe that the the video measures
number is greater (19744) compared to web, upload, download
and latency services. This is explained by the fact when during
a full video test, 30 measurements are captured. In addition,
we remark that more than 93% of the measures are ”OK”,
which means the service is good. This is can be explained on
the one hand, by the quality of the operator’s network in the
Paris region. On another hand, it can be correlated with the
fact that almost the majority of measurements are from LT E
(4G) technology, which ensures good quality. Concerning the
mobility patterns.
IV. CROWDSOURCED NETW OR K MEASUREMENTS
(CNMS)IMPLEMENTATION
Using the 5Gmark,211 parameters (cell/wifi data) can
be collected1. To choose the parameters to be analyzed, we
relied on the study [7] of the literature, where the authors
describe QoS/QoE monitoring tools and related performance
parameters which have an influence on user satisfaction. From
this study, a classification of the parameters on three categories
is achieved including (i) application category (depending on
the service like bit rate video for the video service), (ii) trans-
port and network category (packet loss ratio, network delay,
congestion period, network throughput, etc.) and (iii) physical
category (SNR, symbol/bit error probability, energy efficiency,
etc.). In this paper, we focus on the video application (Figure
2) and LTE technology to study the failed service and the
timeout status of transport connections of the videos.
To that end, we analyze one application parameter (bit rate
video) and a set of physical parameters that we summarize in
Table I.
QoE IF Description min mean max
bitrate
(kbit/s)
Bits number that are conveyed
or processed in a second
12 31398 84679
LT ERSRQ Reference Signal Received
Quality
−20 −10 −3
LT ERSRP Reference Signal Received
Power
−139 −92 −44
LT ERSSN R Reference Signal Signal to
Noise Ratio
−200 97 300
LT Estrength Received signal strength −113 −78 −51
status Test status: OK,
T I M EO U T ,F AI LLED
///
TABLE I
DESCRIPTION OF THE SELECTED PARAMETERS.
The objective of this paper is to evaluate the correlation
between the bit rate video (bitrate) and physical parameters,
like LT ERSRQ and LT Estreng th in order to assess their
impact on the QoE. The considered scenario consists of
YouTube video streaming service with clips of 30 seconds
duration and HD (720x1080) resolution. In order to do this,
we followed the methodology presented in Figure 3-A, that
details the implementation of our CNMA solution via flow
charts.
In a first step, our approach uses the the raw data stored
in CNMA storage module. In this step, we use three filters
that concern (I) video as service, (ii) 4G (LTE) as technol-
ogy and (iii) {bitrate,LT ERSRQ ,LT ERSRP ,LT ERS SN R,
LT Estrength }as parameters. The second step is dedicated
to the labeling process, where the test status values OK,
F AI LLED, and T I ME OU T are encoded 3,2,1respec-
tively. In the third step, we extract the measurements with
low bit rates video (≤5000 Kbits) only. Therefore, 1778
examples remained to build our decision tree. The idea is to
analyze the problems that yield to the low bit rates video (user
dissatisfaction case) and to study the behavior of the physical
parameters in this case.
1https://my.5gmark.com/exports/tests/lexicon

Fig. 3. Description of the performed scenarios
At the end of the 3rd steps (CNMA batch module), the
dashboard module is ran to analyze and to display two kinds of
results : correlation matrix in Figure (4) and graphical decision
tree classifier in Figure (5).
Fig. 4. Correlation matrix
The Figure (4) illustrates an overview of the considered
factors (Table I) as well as their correlation using the Pearson
”r” value. From this figure, we observe that the correlation rate
between the status (service state: F AI LEED = 1,T I M E
OU T = 2 or OK = 3) and the parameters selected in this
study is different. Indeed, the state of the service (status)
correlates more with the bit rate of the application with a
positive rate of 84%. This confirms the results of studies in
the literature which indicate that the video bit rate is one of
the most important parameters of the video streaming service.
Among the physical parameters, it is the RSRQ that correlates
the most with the status of the video service received (status)
with a positive correlation rate of 23%.
Based on the best correlated 3parameters (bit rate video,
RSRQ and signal strength) according to the correlation matrix
(Figure (4)), we plot another kind of plot. This kind consists
on visualizing the relation between these three parameters
by representing the staus values according to them on three
dimensions (3D) plot as presented in the Figure 5.
Fig. 5. 3D plot of the relation between bit rate video (kbits/s), RSRQ and
signal strength
Figure (5) shows that the two video service states (OK
and T I ME OU T ) depend on the bit rate video, where the
video streaming service is normal (status OK) when the bit
rate video exceeds approximately 2000 (kbits/s) and the video
service status is in T I ME OU T when it is approximately
between 900 (kbits/s) and 1600 (kbits/s). Concerning the
F AI LLED state, it depends more on the ”LTE Strength”
physical parameter, where the video service is F AILLE D
status when approximately ”LTE Strength” values are be-
tween the values −50 to −68 according to our dataset.
To confirm these results, we use a decision tree classifier in
the CN MA second module provide the graphical presentation
shown in Figure (6).
CONCLUSION AND FUTURE WORKS
In this exploratory work, the mobile Internet in Paris reagion
(Ile-De-France) was evaluated with the help of Crowdsourced
Network Measurements (CNMs). Based on a the collected
dataset, we have implemented a toolkit, called Crowdsourced
Network Measurements Analyzer (CNMA), for processing,
analyzing and plotting the impact of physical parameters
(like LT ERSRQ ,LT ERSRP ,LT ERSS N R) on the bad per-
formance of YouTube HD video streaming service. We fur-
thermore provide a data set with LTE traces containing more
than 60000 samples. In order to allow deeper analysis of the
data, more traces with others providers are needed.
This preliminary work has led to a lot of perspectives. The first
one consists on the CNMA implementation as an open-source
Web-based application. Knowing that the 5Gtechnology has
just lunched in Paris in March 2021, it can be interesting, as
a second perspective, to achieve a 5GCNMs and CNMA tools
shared with the research community. The third perspective
consists on the implementation of a video subjective QoE

Fig. 6. Correlaton matrix
assessment model like [8] and studying the behavior of the
physical parameters on real subjective video Mean Opinion
Score (MOS). For this perspective, we need a real QoE dataset
that contains MOS scores and bit rate video as features.
REFERENCES
[1] V. Vasilev, J. Leguay, S. Paris, L. Maggi and M. Debbah, ”Predicting
QoE Factors with Machine Learning,” 2018 IEEE International Confer-
ence on Communications (ICC), Kansas City USA, 2018, pp. 1–6.
[2] A. Schwind, F. Wamser, T. Hossfeld, S. Wunderer, E. Tarnvik and
A. Hall, ”Crowdsourced Network Measurements in Germany: Mobile
Internet Experience from End User Perspective,”Broadband Coverage
in Germany; 14. ITG Symposium, Berlin, Germany, 2020, pp. 1–7.
[3] International Telecommunication Union-Telecommunication (ITU-T)
Recommendation : Recommendation ITU-T E.812 – Amendment 1:
Crowdsourcing approach for the assessment of end-to-end quality of
service in fixed and mobile broadband networks. Swaziland, Geneva,
2020, pp. 1–38.
[4] L. Amour and A. Dandoush. Crowdsourced Network
Measurements Analyzer (CNMA) source, 2021. available online:
https://gitlab.com/ESME Git/cnma-analyzer-and-lte-data/-/tree/master/CNMA analyzer.
[5] L. Amour and A. Dandoush. LTE measurments dataset
- Paris (Ile-De-France), April 2021. available online:
https://gitlab.com/ESME Git/cnma-analyzer-and-lte-data/-/tree/master/data.
[6] B. Meixner, J. W. Kleinrouweler, and P. Cesar. 4G/LTE channel quality
reference signal trace data set. In Proceedings of the 9th ACM Multi-
media Systems Conference (MMSys ’18). Association for Computing
Machinery, New York, NY, USA, 2018, pp. 387–392.
[7] W. Robitza, S. Zadtootaghaj, A. Asan , A. A. Barakabitze, E. Grigoriou,
eds. Innovative Quality Of Experience maNagement in Emerging mulTi-
media services: D5.1 QoS/QoE Monitoring Tools. QoE-Net report. Mai,
2016, pp. 1–6.
[8] D. Raca, D. Leahy, C. J. Sreenan and J. J. Quinlan Beyond Throughput:
The Next Generation a 5G Dataset with Channel and Context Metrics.
ACM Multimedia Systems Conference (MMSys ’20), Istanbul, Turkey,
June, 2020, pp. 1–6.
[9] J. Joskowicz, J.C. L´
opez-Ardao,M.A. Gonzalez Ortega, C.L. Garc´
ıa. A
Mathematical Model for Evaluating the Perceptual Quality of Video. In:
Mauthe A., Zeadally S., Cerqueira E., Curado M. (eds) Future Multime-
dia Networking. FMN 2009. Lecture Notes in Computer Science, Vol.
5630. Springer, Berlin, 2009, pp. 1–12.