03/03/2022 05:00
Markus Mandau and Wolfgang Pauler
The toughest and largest independent network test in Austria is entering the next round. For the CHIP network test, the testers traveled all over Austria for two weeks, covering almost 7,880 kilometers with special measuring vehicles, in trains and on foot. They ran hundreds of thousands of individual tests.
The dealer may pay a commission for links on this page, eg for links marked with or green underlining.
More info.
A good cell phone network test needs good partners
Once again, together with our measurement partner NET CHECK, we are setting the standards for an independent and transparent test of mobile networks in Austria.
The biggest difference to the previous tests: The availability and performance of the 5G networks are determined with much more effort and are given more weight in our rating.
Our partner NET CHECK carried out the tests. The specialist in evaluating, analyzing and improving mobile networks used a total of three test vehicles with the "Diversity Benchmarker II" test systems from Rohde & Schwarz Mobile Network Testing (MNT) this year. Two of them drove through Austria to test the quality of the "classic" mobile phone networks. In addition, another vehicle drove through the country with S21 smartphones activated for 5G, specifically to test the performance and availability of the latest, fifth generation of mobile communications.
During the tests on the train and on foot, another team was on the road to test mobile Internet access everywhere a measuring vehicle cannot reach: on long-distance trains, in local public transport and in heavily frequented places in major cities, such as train stations, shopping centers or airports. The team was equipped with two "Freerider III" model backpack test systems from R&S MNT – by far the largest manufacturer of mobile communications benchmarking systems. There were also scanners everywhere, which recorded whether, and if so, on which frequency and with which bandwidth the network operator's 5G signals were available wherever our teams were traveling.
Tirelessly on the go: NET-CHECK engineers covered almost 7,900 kilometers in Austria this year for the CHIP network test.
CHIP STUDIOS
Drive test: Smartphones belong on the roof
However, the main work was done by a total of 30 smartphones that were in operation at the same time – ten for each of the three network operators. Galaxy S21+ 5G models were used for the telephony tests and for our internet and 5G performance tests.
When making calls (VoLTE, EVS), the S21+ 5G support all technologies actively used in the Austrian networks. The same applies to mobile internet access including 5G. Each drive or walk team used three S21+ 5Gs per network operator: A pair of S21+ 5Gs tested the telephony quality of the respective mobile network provider with mobile-to-mobile calls and WhatsApp calls, with another S21+ 5G we tested for every network operator the performance of mobile Internet access. The only exception to this rule: no telephony measurements were taken during the 5G test drive; Austrian mobile networks do not currently support Voice over New Radio, as telephoning via 5G is called.
For our tests, we installed nine smartphones in a specially designed roof box for each vehicle, and they were checked by a multi-channel measuring system in the trunk. The overall "Diversity Benchmarker II" system can also be used in extreme weather conditions, because it offers a separate, air-conditioned box for each smartphone, which is made of a special plastic that does not dampen the mobile phone signal. This is important, because during the test we relied entirely on the integrated antennas of the smartphones and not on external antennas attached to the vehicle. In this way, we were able to ensure that the multi-antenna LTE radio technology MIMO is tested under realistic everyday conditions.
Route: The measurement teams traveled 7,660 kilometers in several cars and by train throughout Austria. They visited a total of 35 cities.
CHIP
Cell phone network test from Vienna to Innsbruck: Here we tested
We sent the measuring vehicles separately on motorways and country roads throughout Austria and also drove on secondary roads through sparsely populated areas. The NET-CHECK experts covered more than 4,350 kilometers in two cars for the CHIP network test to check the 4G technology.
Another team separately drove 2,670 km through Austria in a car to check the status of the 5G network. They ran the performance tests for our 5G score. Only the measurements that took place via the "fast 5G" (at 3.6 GHz) are taken into account here – everywhere else 5G does not bring any significant performance advantages compared to 4G.
The test route led across Austria. Our test teams spent a particularly large amount of time in five major cities (Vienna, Graz, Linz, Salzburg and Innsbruck). But we also made more detailed test drives in ten other medium-sized cities and another 20 small towns selected by the CHIP editors shortly before the start of the measurement. In addition, two measurement engineers with backpack systems traveled a good 640 kilometers by train to measure how well telephoning and mobile surfing worked on long-distance trains. If available, they went to so-called “mobile phone compartments” that promise a reasonable supply of mobile communications.
In the five major cities mentioned above, the measurement technicians then got out and visited heavily populated places on foot and by public transport. In the inner cities, they measured how well the networks still work in particularly popular locations when they have to supply many users at the same time.
5G measurements – the networks really do that
The network operators are currently bringing their 5G network to the customer in different ways: Magenta and A1 (also Drei to a very small extent) offer 5G together with 4G in the same frequency band (Dynamic Spectrum Sharing, DSS), also with a very small available bandwidth. This strategy allows 5G to be made widely available quickly. The downside: The 5G performance is not significantly faster than with a pure 4G offer.
All three network operators also use the auctioned licenses for the frequency range at 3.6 GHz, on which they exclusively offer 5G, with much higher bandwidths. This allows very decent data rates of over one gigabit per second to be achieved.
For our 5G rating, we assess the availability of the 5G networks along our measurement route. A special scanner from R&S MNT (TSME6) recorded the band and level at which 5G signals could be received along a route of around 4,350 kilometers. From this, we then determine what percentage of the measurement section was covered with 5G. For the evaluation, we differentiate between the geographical coverage with fast 5G at 3.6 GHz in the cities and the geographical availability of 5G overall (3.6 GHz + DSS frequencies) in the cities and on the connecting roads on which we used during the measurements.
We measured the performance of 5G with several Samsung Galaxy S21+ 5G. The smartphones were housed in the roof box.
CHIP STUDIOS
5G performance
During the 1,520-kilometer drive, we also examined the performance of the 5G network at 3.6 GHz. We deliberately avoided speed measurements in the "slow" 5G network that is offered via DSS.
Half of the overall 5G rating is made up of the performance rating and half of the availability rating.
Telephone quality in the cell phone network test
Even if users make fewer calls on average than in recent years, a good mobile network must allow calls to be made reliably and in good quality. In addition to "normal" calls via the mobile network, we also check WhatsApp conversations, since these are now just as normal for many users as "normal" conversations. However, there are more problems here than with normal telephoning. The reason: Calls with WhatsApp & Co. are treated like normal data traffic on the network; This means that they are not given a guaranteed data rate and are not forwarded preferentially by the routers and other network elements – these privileges are reserved for normal telephone calls.
For the evaluation of the WhatsApp calls, we determine similar parameters as for the evaluation of conventional telephone calls: We look at the success rates when establishing a call, whether the test telephone calls remain stable over the entire duration of 2 minutes or break off beforehand and how good the voice quality is. If possible, these cell phones used LTE to call each other, otherwise they used 2G or, in very few cases, 3G.
If successful, each phone call lasted two minutes, with the two smartphones exchanging several German test phrases alternately in each direction – just like in a real conversation. Their quality was automatically evaluated using the standardized POLQA algorithm, which our measurement partner R&S MNT played a leading role in developing. POLQA estimates a so-called Mean Opinion Score (MOS), which ideally is just under 5 points; the worst possible value is 1.
We use broadband speech samples, thanks to which we can evaluate the speech quality in a very differentiated manner. If available, the smartphones may use the "Enhanced Voice Services", EVS. The EVS codec processes almost the entire frequency spectrum that can be heard by the human ear. Speech and music sound much more natural than when using narrower codecs, such as those used with HD Voice.
In practice, it has been shown that test sentences with a MOS below 1.6 show serious communication problems on the part of the interlocutors. When evaluating the voice quality, we therefore pay attention to which proportion of the voice samples were rated with an MOS below 1.6; however, we also look at the average MOS score of each vendor.
Even more important to us than the voice quality is that the connection is successfully established at all and that it is then also stable (altogether 63 percent of the telephony grade). After all, nothing is more annoying than a conversation that keeps getting interrupted. We award the remaining points of the telephony rating for the call setup time for classic voice calls. We define the call setup as it is experienced by the customer in practice: it goes from pressing the "Call" button to the point at which the smartphone signals that the call has been successfully set up and the call can begin. Here we evaluate the average call setup time per provider. More important to us, however, is how often it takes more than 15 seconds for a call to be put through.
Samsung Galaxy S21+ 5G in the roof box: We use "our" test smartphone for all tests on telephoning and mobile Internet access, because it supports VoLTE, EVS and LTE Cat.20.
CHIP STUDIOS
How good are cell phone networks for mobile Internet?
We used Samsung's top models, the Galaxy S21+ 5G, for the mobile Internet measurements. They support channel bundling (carrier aggregation), high-quality modulation methods and multi-antenna systems (MIMO).
In the upload and download tests, the smartphones had the task of sending and downloading as much data as possible to the server within 10 seconds. In addition, we performed full uploads and downloads of large files to evaluate the success rate. Since we wanted to exploit the full capacity of the networks, we also carried out download measurements, in which ten parallel data connections to the server were established.
The success rate and the average data rate over the measurement period were evaluated in the upload/download tests. We paid even more attention than before to how well the bottom 10 percent of our measurements were for each provider. They are a measure of how good the basic supply of the network is, which is why they determine two-thirds of the result of the speed tests.
Reliability also plays an important role when surfing the web: If a test website was not completely displayed after a specified period of time, we aborted the test and rated it as unsuccessful.
On the go with a measurement backpack: Our testers covered long distances on foot to test the mobile Internet access.
CHIP STUDIOS
On-site verification: Our tests pose real challenges for mobile networks.
CHIP STUDIOS
Network test Austria 2022: The overall rating
Our overall result is based on the four major rating categories "Mobile Internet" (47.5 percent contribution to the final score), "Telephony" (35 percent) and 5G (17.5 percent). You can see the detailed results in our main article.
