Bandwidth management firm Sandvine issued a report Monday claiming that Netflix’s ISP Speed Index and other “over-simplified” reports and benchmarks are misleading and do not offer an accurate measurement of Internet quality.
“Commonly-trusted quality benchmarks fail to deliver a true picture, while economic competition affects service delivery much more than we may think,” Sandvine concluded in the report, titled: “Exposing the Technical and Commercial Factors Underlying Internet Quality of Experience.” “Benchmarks are overly simplistic, some with significant inaccuracies, and purport to focus on a single spot (the access network) while ignoring other highly impactful impairments in the chain,” said Sandvine, which analyzes both wired and wireless broadband networks and counts Comcast among its customers.
For the report, Sandvine said it analyzed several popular benchmarks that are used to determine Internet quality, holding that they don’t provide a true measure of the access network.
Sandvine’s study drilled down on the Netflix’s ISP Speed Index, which presents the average speed of Netflix streams, typically finding that ISPs that are part of Open Connect, Netflix’s private content delivery network tend to rate highly. For example, Google Fiber, Cablevision Systems, Cox Communications and Suddenlink Communications, which happen to be Open Connect members, have routinely represented the top four U.S. ISPs in Netflix's monthly ratings, including in the most recent August results. According to Netflix’s commercial policy, only customers of ISPs that are part of Open Connect can access to Netflix’s library of “Super HD” titles, which require at least 5 Mbps. And only certain types of devices and models that can support the 1080p Super HD video streams.
Because not all Netflix subscribers are allowed to access the company’s highest resolution streams, the Netflix ISP Speed Index “is not an accurate measure of capacity,” because it shows “average video bitrate” rather than “peak speed,” Sandvine argued, noting that the average bit rate of Super HD titles is roughly 12 Mbps, while standard-definition Netflix streams clock in at 2 Mbps to 3 Mbps. Therefore it’s possible that an ISP that is part of Open Connect with access to more highly encoded Super HD titles will fare better in the rankings than non-Open Connect ISPs that don’t use Netflix caching devices.
“From this we conclude that the ‘ISP Speed Index’ is largely showing the amount of 1080p versus standard definition delivered, and thus is driven by device mix as much as content choice or network peering, but is not reflective of network capacity at all,” Sandvine said in the report. “In all cases, the ‘average capacity’…is greater than the ‘speed index’ shown, thus we conclude Netflix is showing the aggregate ‘demand’ on their service rather than the ‘capacity’ of the access network,” Sandvine added.
Netflix’s “Index doesn’t measure ISP ‘quality’ at all, a fact that is not obvious from the Index’s presentation and is poorly understood by Index followers,” Sandvine noted in this blog post that highlights the 35-page report’s Netflix findings.
Thus, Sandvine suggests that Netflix would need to break out device type and “allowed” stream speed” to increase the usability of its ISP Speed Index, but acknowledges that Netflix has a “vested interest” in these reports because it can help drive its Open Connect peering arrangements with ISPs.
Netflix’s ISP Speed Index wasn’t the only target of the Sandvine Internet quality report.
Sandvine said the YouTube benchmark likewise does a better of measuring demand than network capacity and quality. While YouTube does not restrict access to high-quality streams, benchmark results will skew lower if a user tends to watch only low-definition videos.
For example, the average speed reported by the YouTube benchmark will also rate higher if viewed with a Google Chromecast device on a 1080p TV, versus an older device, such as the Nintendo Wii, which tend to request lower resolution video streams.
Sandvine also took a look at Ookla’s Speedtest, noting that it typically presents two or more servers to a user as “suggested” ones, selected based on geographical proximity estimated by the user’s IP address, resulting in a broad variance of results. “Taking the suggested servers and using them on an otherwise completely idle connection, we find that the two servers consistently give very different results.”
The report also found that nature of adaptive video, a technique used by Netflix and other over-the-top firms to adjust the resolution and bit rate based on the available bandwidth, won’t degrade like other forms of Internet traffic.
Adaptive video, Sandvine said, “may ‘push aside’ non-adaptive content such as gaming, VoIP, and HTTP.”
