Codecs, compression & latency

Chapter 2 showed that a 4K60 picture is about 12 Gbps of raw pixel data. A 1 Gb network port carries less than a tenth of that. To move video over ordinary networks, or over the internet to a far-end meeting room, most systems compress it with a codec (coder–decoder). Every codec trades three things against each other: bandwidth, picture quality and latency. You can usually have two.

How compression saves space

Codecs remove information you are unlikely to notice. They work in two broad ways:

• Intra-frame compression squeezes each frame on its own, much like a JPEG photo. It can start sending a frame before it has finished receiving it, so the delay can be a small fraction of a frame.
• Inter-frame or long-GOP compression also exploits similarities between frames: most of a talking-head video doesn't change from one frame to the next, so only the changes are sent. A group of pictures (GOP) starts with a complete frame and follows it with frames that only describe differences. This saves far more bandwidth, but the encoder has to look across several frames, which adds latency.

Latency: the delay you can feel

Latency is the time from light entering a camera, or a frame leaving a laptop, to the picture appearing on a display. It builds up along the whole chain:

Different jobs tolerate very different amounts:

• Interactive use — moving a mouse on a laptop shown on the room display — needs latency low enough that the pointer doesn't feel sluggish, ideally well under a frame or two.
• Live image magnification (IMAG) — a presenter shown on a big screen behind them — needs very low latency, or the audience sees their lips move out of time with their voice.
• Conferencing already includes network delay across the internet, so long-GOP codecs are acceptable.
• Streaming and recording can tolerate seconds of delay in exchange for efficiency.

Lip sync

Video processing usually takes longer than audio processing, so sound can arrive before the picture. Viewers are especially sensitive to sound that leads the picture. Most DSPs and many AV devices offer an audio delay setting so the sound can be held back to match the video; set it by watching and listening to someone speaking or clapping on camera.