Networks from Communications Service Providers (CSPs) are complex systems. Between control and user planes, hundreds of metrics can be obtained to evaluate a multitude of performances. Fittingly, network engineers know that no single metric can tell a whole story by itself. When asked which network KPIs are the most critical to keep subscribers satisfied, their answer is inevitably ‘it depends’.
True Quality of Experience (QoE) can only be established by collecting primary subscriber feedback. In the absence of that, current Quality of Service (QoS) calculation methodology generally combines the following KPIs to identify the subscriber experience, from the user plane perspective.
- Connection Setup Success – i.e., if the client-server connection is successfully established all the time before the transfer of data.
- Packet Loss – i.e., Packet Loss during the establishment and/or transfer of data.
- Latency –e., if there are no significant delays in the packet transfer.
- Throughput – i.e., if there is enough bandwidth available to transfer the data.
- Session Duration – i.e., if you do not suffer noticeable hiccups, then you will watch videos, carry calls and play games for longer intervals.
Traditionally, the need to improve those 5 network KPIs has been a rallying cry for business teams and customer experience managers. Partly, they are right: it is true that most digital experiences are negatively impacted when those metrics falter. However, not all experiences are impacted equally: subscribers’ perceptions are shaped differently depending on the different applications they use. As a result, indiscriminate calls for the improvement of those KPIs generally lead to misplaced and suboptimal effort allocations.
Sure enough, increased latencies, increased packet losses and lower throughputs, in one way or another, represent bad QoE. But subscribers using different applications demonstrate different sensitivities to the deterioration of each network KPI. For example:
- The same Latency increases might have a higher shortening impact on Instagram Session Durations than on YouTube Session Durations.
- Packet Loss can prove more damaging to the duration of WhatsApp voice call than to the duration of sessions from other applications.
- Throughput declines might not cause meaningful impact on Instagram browsing experience. At least, not until they cross a threshold that, for YouTube, would have already pushed away even the most persevering of the watchers.
With new and evolving applications, then, the above set of KPIs may not be sufficient to give an accurate indication of the actual subscriber experience.
Let’s consider the following examples:
1. VoIP Calling (e.g., WhatsApp Calls, VoLTE Calls)
2. Video Streaming (e.g., Youtube)
3. Video and Voice Conferencing (e.g., Blue Jeans, Zoom)
4. Online and Cloud Gaming (e.g., Call of Duty Multiplayer, Google Stadia)
Because each traffic type has different characteristics, it must be catered for differently, with a broader set of KPIs to identify the QoE delivered for each. While calculating subscriber experience from the CSPs perspective for each of these 4 examples, the following set of metrics should thus be considered:
1. VoIP Calling: Connection Setup, Call duration, Codec, Jitter, Packet Loss, and Latency.
2. Video Streaming: Initial Buffer Times, Stalls, Streaming Quality, and Resolution Changes.
3. Video and Voice Conferencing: Connection Setup, Call duration, Throughput, Jitter, Packet Loss, Connection Success, and Resolution.
4. Online and Cloud Gaming: Session Duration, Loading Time, Latency, Packet Loss, and Throughput.
Not only different digital experiences call for different sets of KPIs, but CSPs must also consider three elements when evaluating subscriber experiences: QoS Weightage, limitations of the Access Technology, and Connection QoE Parameters.
First, different QoS KPI Weightage should be applied to various traffic types while calculating subscriber experience. For example, Latency and Packet Loss for Online Gaming should have higher precedence over other KPIs for that specific application category. For Video Streaming, Throughput should be prioritized over the Initial Buffer Time.
Second, incorporating the limitations of the Access Technology is another important factor while calculating subscriber experience. The thresholds set for Latency and Bandwidth in 5G or 4G are different compared to the ones in 3G. For instance, watching Youtube videos over a 3G connection cannot provide the same HD experience as it would on 5G – and the tolerance threshold of the watcher for a decreased quality on 3G is higher than on 5G. Practically speaking, an average Latency of 70 ms for 3G connection is considered ‘good’ compared to 4G, where the average can be around 50 ms. Hence the importance of considering the underlying technical limitations of the Access Technology when calculating subscriber QoE for different types of traffic.
Finally, analytics platforms should also identify and consider per flow network parameters configured by the CSP while calculating subscriber experience. The following table provides us the QoS Class Identifier (QCI) and 4G access link parameters recommended for various traffic types.
|
QCI |
Resource Type |
Priority Level |
Packet Delay Budget |
Packet Error Loss Rate |
Example Services |
|
1 |
GBR |
2 |
100 ms |
10-2 |
Conversational voice |
|
2 |
GBR |
4 |
150 ms |
10-3 |
Conversational video (live streaming) |
|
3 |
GBR |
3 |
50 ms |
10-3 |
Real-time gaming; vehicle-to-everything (V2X) messages |
|
5 |
Non-GBR |
1 |
100 ms |
10-6 |
IMS signaling |
|
9 |
Non-GBR |
9 |
300 ms |
10-6 |
Video (buffered streaming); TCP-based (e.g., www, e-mail, chat, File Transport Protocol (FTP), peer-to-peer (p2p) file sharing, progressive video, etc.) |
Being lower priority, Non-Guaranteed Bit Rate (Non-GBR) bearers can suffer higher latency under congestion while GBR bearers are not immune to high latency. For instance, traffic allocated with QCI = 1 is used for handling Voice calls, while QCI = 9 for normal internet browsing. Hence, higher latency in QCI = 1 has more deteriorating impact on subscriber experience than in QCI = 9.
Same goes for 5G, where Ultra-Reliable Low-Latency Communication (URLLC) slice is designed to handle Latency better than Enhanced Mobile Broadband (eMBB), or where eMBB is allocated more throughput per connection compared to massive Internet of Things (mIoT) slice. As such, the Throughput weightage in the subscriber experience score has to be greater in eMBB slice compared to URLLC.
All in all, network KPIs and subscribers’ Quality of Experience do not always mirror each other. Understanding the impact of each one of those KPIs and parameters on the QoE enjoyed by mobile data users when engaging with different popular applications is the first step in delivering superior digital experiences. But most importantly, how can this knowledge be leveraged and how can network analytics technologies help?
At Mobileum, we collect and correlate real-time network data across various dimensions, which we can complement by ingesting data feeds from 3rd party elements, and we further leverage proprietary machine-learning algorithms to provide advanced network analytics, even from encrypted streams. Our deep network analytics capabilities enable us to surface rich and granular insights that take into considerations how subscriber perceptions are shaped differently depending on the different applications they use, and how network KPIs can and should be accounted for when calculating overall subscriber QoE.
In possession of the knowledge that different digital experiences call for different network performances, business teams can demand network improvements that make real sense. By tailoring the right medicine to the right symptoms, and by understanding that different applications respond in unique ways, CSPs can better allocate their time and resources. Neither over- nor under-spending, then, on their network improvement investments.
Let Us Know What You Thought about this Post.
Put your Comment Below.