What A Codec Actually Does To A Frame

A raw video frame is an enormous quantity of numbers, and a codec's job is to describe that frame in far fewer numbers. Understanding what it throws away explains almost every artifact you have ever seen.

Raw frames are impractically large

An uncompressed high-definition frame stores a brightness and color value for every pixel. Multiply that by the pixel count, then by the number of frames per second, and the data rate becomes unmanageable for any consumer connection.

Storage is only half the problem. Moving that many bytes across a home network or a hard drive in real time demands sustained throughput that ordinary equipment cannot supply for long.

So every video you watch has been rewritten into a compact description that a decoder expands back into pixels at playback time. The original numbers are gone and are not recoverable.

The frame is broken into blocks

Encoders divide a frame into rectangular blocks rather than working pixel by pixel. Neighboring pixels usually resemble each other, so a block can be described far more briefly than its individual members.

Each block is transformed from a grid of values into a set of coefficients describing patterns of varying coarseness. Broad, gentle gradients land in a few coefficients while fine texture spreads across many.

This reorganization does not itself save anything. It arranges the data so that the next step can discard the parts a viewer is least likely to miss.

Quantization is where information disappears

The encoder rounds those coefficients, coarsely for fine detail and gently for broad shapes. Rounding many small values to zero collapses them into something a compact code can express in almost no space.

How aggressively it rounds is the single most consequential decision in the whole process. Light rounding preserves texture and costs data; heavy rounding produces the blocky, smeared look of a badly squeezed clip.

Because the discarded values are never written down, no later processing restores them. Every subsequent edit works from the reduced version rather than from the original picture.

Human vision decides what is expendable

Eyes resolve brightness far more precisely than color, so encoders routinely store color at a lower spatial resolution than luminance. Most viewers never notice on ordinary footage.

Vision is also poor at spotting fine detail inside busy texture and unusually good at spotting error in smooth areas. Encoders exploit that asymmetry by protecting flat regions and starving complicated ones.

Those assumptions hold for typical scenes and break for atypical ones. Sharp graphics, text overlays and saturated red objects fall outside the model and tend to look worse than the footage around them.

Decoding is the cheap half

An encoder searches among many possible descriptions to find a good one, which takes real computation. A decoder simply follows the description it is given.

That asymmetry is deliberate, because a file is encoded once and decoded millions of times. Playback hardware can be small and low-powered while encoding stays a heavier task.

It also explains why a phone plays a demanding file smoothly yet takes minutes to export a short clip. The two operations are not comparable in effort.