What Happens To Audio When Video Is Compressed

The soundtrack of a video is compressed separately from the picture and by very different means. What it discards is chosen using a model of hearing.

Masking decides what can be removed

A loud sound makes nearby quieter sounds inaudible, both at similar frequencies and in the moments just after. Anything hidden this way need not be stored.

The encoder splits the signal into frequency bands and estimates, band by band, how much error each can hide. Loud bands can absorb more error than quiet ones.

The savings come from storing hidden components with very little precision, or not at all. The reconstruction is not the original waveform.

Bitrate determines how much must be discarded

At a generous bitrate the encoder discards only what genuinely falls below the masking threshold, and most listeners hear no difference.

As the budget tightens it must discard audible content. It sacrifices the highest frequencies first, then stereo separation, then precision in the midrange.

The characteristic low-bitrate sound is dull, narrow and slightly watery. That last quality comes from time-domain smearing of brief sounds.

Some material defeats the model

Applause, rain and cymbals consist of dense, brief, unpredictable events. The masking model handles them poorly and the result develops an artificial swirling quality.

Solo instruments in near silence are hard for the opposite reason. Nothing masks the coding error, so it is heard directly as a faint warble.

Speech in a quiet room falls into the same category, which is why dialogue can sound processed on heavily compressed streams.

Repeated compression compounds the damage

Each pass discards what its model deems inaudible, and a second encoder analyzes the already-damaged result. It may keep artifacts and discard remaining real content.

Different codecs make different assumptions, so converting between them is worse than repeating the same one. Their models disagree about what mattered.

Editing from a compressed source and re-exporting adds a generation every time. Keeping an uncompressed or lightly compressed master avoids the accumulation.

Audio gets a small share of the total budget

In a typical video file the soundtrack occupies a small fraction of the data. Increasing its allocation costs little in file size.

Yet audio problems are often more noticeable than picture problems, since viewers tolerate a soft image far better than harsh or unintelligible sound.

Platforms re-encode uploaded audio to their own specification regardless, so supplying high-quality audio at upload matters more than matching their final settings.