Streaming video quality: our testing methodology and findings
When Netflix updated its VMAF repository in June 2026 with a new generation of v1 perceptual models, it underscored a reality the streaming industry has been slow to acknowledge publicly: the tools for measuring video quality are evolving faster than the average viewer's ability to perceive the difference — and the gap between what a platform advertises and what actually reaches a living-room display remains stubbornly difficult to close. Resolution labels like "4K UHD" and "Full HD" have become marketing shorthand, but the underlying picture quality depends on a chain of technical variables that most subscribers never see and most platform interfaces never expose.
Understanding how streaming video quality is tested — and where those tests fall short — matters for anyone making decisions about which service deserves a monthly fee, which device to connect, and whether a broadband plan genuinely supports the experience being sold.
Beyond Resolution: Why Bitrate and Codecs Define Your Picture
The most persistent misconception in streaming is that resolution equals quality. A 1080p stream encoded at a generous bitrate with an efficient codec can look materially better than a 4K stream starved for bandwidth and compressed with older tooling. The reason is straightforward: resolution describes the number of pixels on screen, but bitrate determines how much visual information each frame actually carries. A 4K frame contains four times the pixel count of 1080p; if the encoding budget does not scale proportionally, the encoder must discard detail, introduce artifacts, or both.
Codec selection compounds the issue. The transition from H.264/AVC to H.265/HEVC — and, increasingly, to AV1 — allows platforms to deliver equivalent perceptual quality at lower bitrates, but only when the playback device hardware supports decoding the newer codec natively. When it does not, the player falls back to an older, less efficient stream, and the viewer sees a degraded picture without any indication of why.
Adaptive bitrate (ABR) players add another layer of complexity. These algorithms select among multiple resolution and bitrate representations — known as an encoding ladder — based on estimated throughput, buffer occupancy, and the display device's capabilities. The DASH Industry Forum's reference implementation, dash.js, exposes the decision factors: current buffer level, selected representation bitrate, resolution, frame rate, dropped-frame counts, abandoned segment requests, and recent switch history. Each of these variables can shift mid-playback, meaning the quality a viewer experiences in the first minute of an episode may differ substantially from what they see twenty minutes later, even on the same connection.
Netflix's published connection-speed recommendations illustrate the point without resolving it. The service advises a stable connection of at least 3 Mbps for 720p HD, 5 Mbps for 1080p Full HD, and 15 Mbps for 4K UHD. These figures describe minimum thresholds under favorable conditions — they are not guarantees of delivery bitrate, nor do they account for codec efficiency, content complexity, or the encoding ladder a particular title uses. A viewer on a 25 Mbps connection may still receive a 1080p stream if the ABR algorithm detects instability, congestion, or device limitations.
Resolution is a container. Bitrate and codec efficiency determine what fills it — and those two variables shift constantly during every stream you watch.
The Role of Objective Metrics: VMAF and ITU-T Standards
To move beyond subjective impressions and marketing claims, the industry relies on a set of objective quality metrics — mathematical models that attempt to predict how a human viewer would rate a piece of video. The most prominent of these is Netflix's Video Multimethod Assessment Fusion, or VMAF, an open-source full-reference metric that compares a distorted (compressed) video against an uncompressed reference and produces a score on a 0-to-100 scale.
VMAF's default model was trained under specific viewing assumptions: a 1080p television display, a viewing distance of three times the screen height, and distorted videos rescaled to 1080p before comparison. These parameters matter. A VMAF score computed under the default model does not directly describe the experience of watching a 4K stream on a 65-inch display from six feet away, nor does it account for a phone screen held at arm's length. The score is a prediction, not a measurement of perception, and its accuracy depends on how closely the test conditions match the model's training assumptions.
The VMAF repository also bundles implementations of several other metrics — PSNR, PSNR-HVS, SSIM, MS-SSIM, and CIEDE2000 — each of which captures different aspects of visual fidelity. PSNR measures pixel-level error; SSIM evaluates structural similarity; CIEDE2000 assesses perceptual color difference. No single metric captures the full picture, which is why serious quality assessments typically report multiple scores.
At the standards level, the International Telecommunication Union provides two foundational recommendations. ITU-T P.910, approved in its 2023 edition, defines non-interactive subjective video-quality assessment methods: absolute category rating, degradation category rating, and pair comparison, along with reference conditions, experimental design, and viewing-environment specifications. These are the protocols for running controlled human-viewing tests — the gold standard against which objective metrics are validated.
ITU-T P.1203, approved in 2017, takes a different approach. It defines an objective model for estimating the quality experienced by end users of progressive-download and adaptive audiovisual streaming. Unlike VMAF, P.1203 accounts for factors beyond compression artifacts: initial loading delay, stalling caused by rebuffering, frame-rate variation, spatial rescaling, and media adaptations. It is designed to model the real-world playback experience, not just the encoded file.
Together, these standards form the methodological backbone of any rigorous streaming-quality assessment. But they describe frameworks, not answers — the quality of the output depends entirely on the discipline of the test design.
Capturing Real-World Performance Through Playback Telemetry
Laboratory metrics computed on encoded files tell only part of the story. What actually reaches the viewer's eyes is shaped by the playback pipeline: the device hardware, the operating system, the application or browser, the display panel, and the network conditions at the moment of streaming. Capturing this real-world performance requires telemetry — structured data extracted from the player during live playback.
A well-instrumented test logs, at minimum, the following fields at regular intervals: current buffer level in seconds, the bitrate of the selected representation, the decoded resolution, the frame rate, and a bitrate estimate derived from decoded video bytes over the measurement interval. These fields, drawn from the DASH monitoring specification, reveal what the ABR algorithm is actually doing rather than what it is theoretically capable of doing.
The browser's Media Capabilities API, available across major browsers since January 2020, adds another diagnostic layer. Given a video configuration — codec, width, height, bitrate, and frame rate — the API reports whether the configuration is supported, whether playback is expected to be smooth, and whether it is power-efficient. This allows testers to distinguish between a device that cannot decode a given stream and one that can but is choosing not to for efficiency reasons.
Startup delay — the time between a play request and the first frame appearing on screen — and rebuffering events (stalls) are equally critical. A stream that delivers pristine 4K frames after a twelve-second loading screen and two mid-episode interruptions delivers a worse experience than a stable 1080p stream that begins instantly and never hitches. ITU-T P.1203 incorporates these factors precisely because they dominate viewer satisfaction in ways that a VMAF score computed on a static file cannot capture.
A VMAF score on an encoded file is a photograph of a car in a showroom. Playback telemetry is the odometer, the fuel gauge, and the road conditions — the data that tells you how the drive actually feels.
Controlled Subjective Assessment and Viewing Environments
Objective metrics, however sophisticated, remain proxies for human perception. The ITU-T P.910 standard exists because the industry recognizes that the final arbiter of quality is a person watching a screen. Controlled subjective tests — conducted under specified lighting, with calibrated displays, at defined viewing distances, using trained or naive panels depending on the study design — produce mean opinion scores (MOS) that capture perceptual judgments no algorithm can fully replicate.
The challenge is scale. A subjective test involving dozens of participants evaluating hundreds of clips under controlled conditions is expensive, slow, and difficult to reproduce across different labs. A 2024 open-access dataset containing 700 video sequences encoded in H.264 and H.265/HEVC, assessed using subjective MOS alongside SSIM and VMAF under varying bitrates, resolutions, and packet-loss conditions, represents a significant contribution to the field — but the publication itself notes that no standardized mapping function exists between subjective and objective results. A VMAF score of 85 does not translate to a MOS of 4.2 on a five-point scale with any universal reliability; the relationship depends on content type, encoding conditions, and the specific subjective test protocol used.
This gap has practical consequences. When a platform claims "near-transparent quality" or a review outlet assigns a numerical picture-quality rating, the underlying methodology determines whether those claims hold up. A test that computes VMAF on server-side encodes without capturing playback telemetry, without documenting the device and display, and without controlling for network variability is measuring the encoding pipeline — not the viewer experience.
Navigating the Limitations of Universal Quality Benchmarks
The honest conclusion from the available methodology is sobering: no single number, metric, or test can definitively rank streaming services by visual quality across all titles, devices, codecs, and network conditions. The variables are too numerous, the interactions too complex, and the viewing environments too diverse.
What a rigorous methodology can do is establish controlled comparisons — the same title, the same device, the same network profile, measured at the same time — and report the results with full transparency about the test conditions. It can document which codec was delivered, at what bitrate, at what resolution, with what buffer behavior and startup delay. It can compute VMAF and other metrics against identified reference files using a named model version and software build. And it can acknowledge, explicitly, where the data ends and where interpretation begins.
For the viewer making a subscription decision, the practical takeaway is this: advertised resolution is the least reliable indicator of streaming quality. The factors that actually determine the picture on your screen — encoding efficiency, ABR behavior, device support, network stability — are largely invisible in platform marketing and require either technical instrumentation or disciplined comparative viewing to assess. Until platforms provide transparent quality dashboards — showing the delivered bitrate, codec, and resolution in real time, as some already do in developer-facing tools — the gap between what is promised and what is delivered will remain a matter of informed estimation rather than settled fact.
The tools to close that gap exist. ITU-T P.1203 and P.910 provide the standards. VMAF and its companion metrics provide the computational framework. Playback telemetry and the Media Capabilities API provide the real-time diagnostics. What is missing is the industry commitment to apply them consistently, publish the results openly, and let the data — not the marketing — set the expectations.
