How Is the Audio Track Stored in a Video File? Demuxing and Stream Copy Logic

What happens inside the file when you extract audio from a video? The audio isn't "separated" — it's already sitting there as a separate track, just packaged interleaved with the video data. This article explains containers' internal structure, how tracks are stored, and the technical difference between stream copy and re-encoding.

How tracks sit inside a container

If you opened a video file and looked inside, you wouldn't see a tidy structure like "all the video first, then all the audio." Instead you'd find a stream of small packets interleaved:

[video packet 0.00s][audio packet 0.00s][video packet 0.04s]
[audio packet 0.02s][video packet 0.08s][audio packet 0.04s]...

That's called multiplexing. The reason is playback efficiency: while watching a video you need both the picture and the sound of a given moment at the same time. If they were stored at opposite ends of the file, the player would have to keep seeking back and forth. When streaming over a network that would be a completely unusable structure.

Demux (demultiplexing) is the operation of separating that interleaved stream and gathering each track on its own. That's the first step of audio extraction.

The critical point: demuxing involves no decoding at all. It reads the packets, looks at which track they belong to, and separates them. It never looks inside the audio data.

Track structure and metadata

Every container has a structure holding information about the tracks. In MP4 that's the trak structures inside the moov atom; in MKV it's the Tracks element.

The information stored for each track:

| Information | Meaning | |---|---| | Track type | Video, audio, subtitle, metadata | | Codec identifier | AAC, H.264, AC3, Opus... | | Timescale | The unit of the timestamps | | Duration | The track's length | | Language | English, Turkish... | | Channel count | Mono, stereo, 5.1 | | Sample rate | 44100, 48000 Hz | | Bitrate | The average data rate | | Codec-specific data | The initialization parameters the decoder needs |

That last row matters: many codecs need a configuration block before they can start decoding. For AAC that's called AudioSpecificConfig and it carries data like sample rate, channel layout and profile. That block is stored in the track metadata, not inside the audio packets.

During audio extraction that configuration block also has to be transferred correctly to the new file — otherwise the resulting file won't play or will play at the wrong speed.

Stream copy: the lossless route

In a stream copy operation, this happens:

1. The container is demuxed and the audio packets are separated.
2. The packets are never looked inside. The encoded data stays as it is.
3. A new container (or raw file) is created.
4. The packets are written into the new container along with their timestamps.
5. The track metadata and codec configuration are transferred.

The result: the audio data is bit-for-bit identical. There's no quality loss at all. The operation runs at file copy speed because no computation is done.

The condition: the target file format has to be able to hold that codec.

| Source audio codec | Stream-copyable targets | |---|---| | AAC | .m4a, .aac, .mp4, .mkv | | MP3 | .mp3, .mkv, .mp4 | | Opus | .opus, .ogg, .mkv, .webm | | Vorbis | .ogg, .mkv, .webm | | FLAC | .flac, .mkv | | PCM | .wav, .mkv, .mov | | AC3 / DTS | .ac3, .dts, .mkv |

The practical consequence of that table: when extracting audio from your MP4 videos, choosing M4A as the target means a stream copy and no quality loss at all. If you want the same content as MP3, re-encoding becomes mandatory.

Re-encoding: the lossy route

If the target format is incompatible with the source, a two-step operation is needed:

1. Decoding. The encoded audio packets are decoded to PCM (a raw sample array).

2. Re-encoding. That PCM data is compressed from scratch with the target codec.

The problem when going from a lossy source to a lossy target is that the information the first encoder discarded is already gone, and the second encoder discards information again according to its own psychoacoustic model. Two different sets of "discards" stack up and the loss accumulates.

On top of that, the quantization noise left by the first encoding looks like "real audio" to the second encoder, which spends bit budget trying to preserve it.

That's why the target bitrate should be kept one notch above the source's — it doesn't eliminate the loss but limits it.

The exception: if the source is lossless (PCM or FLAC), re-encoding adds only one loss and that's a normal encoding operation. In that case the quality depends entirely on the bitrate you choose.

Timestamps and sync

Every packet carries a timestamp. Actually two:

PTS (Presentation Timestamp): when this packet gets displayed/played.

DTS (Decoding Timestamp): when this packet needs to be decoded.

For audio the two are usually the same. For video they can differ — because B-frames predict from later frames, the decoding order differs from the presentation order.

In audio extraction, timestamps matter for two reasons:

Start offset. In some recordings the audio track starts at a different time from the video. In the container that's compensated with an edit list or an offset value. If that information isn't transferred during extraction, the resulting audio ends up shifted a few hundred milliseconds from the video's.

Gaps. In some recordings (especially streams taken over a network) there can be gaps in the audio packets. Stream-copying carries those gaps across; some tools fill them, some ignore them and the duration comes out shorter.

The variable frame rate problem

Screen recording software and some phone cameras use variable frame rate (VFR): they save space by dropping frames when there's no motion.

Audio, meanwhile, always flows at a constant rate — 48000 samples/second, without exception.

That asymmetry complicates time calculations in VFR videos. Some tools compute the video's average frame rate, assume it's constant, and interpret the audio timestamps accordingly. The result is a drift that grows as the video goes on.

Audio extracted from an hour-long screen recording can be several seconds off toward the end. The fix is converting the video to a constant frame rate first.

Multiple audio tracks

The MKV container can carry an unlimited number of audio tracks. In a typical film file:

• Track 1: Original language, 5.1 AC3
• Track 2: Dubbed version, 2.0 AAC
• Track 3: Director's commentary, 2.0 AAC
• Track 4: Audio description

Each track carries its own language tag, channel layout and codec.

Extraction tools generally take the default track or the first track. If you want a specific track, the tool has to offer track selection.

The MP4 container supports multiple audio tracks too, but it's used less in practice and some players only see the first one.

Channel mixing

Film audio is usually 5.1 or 7.1 channels. Extracting that into a stereo file involves a downmix: six channels reduced to two.

The standard formula is roughly:

• Left = Front Left + 0.707 × Center + 0.707 × Rear Left
• Right = Front Right + 0.707 × Center + 0.707 × Rear Right
• LFE (the bass channel) is usually discarded or added at low weight

That mixing sometimes causes trouble: in film audio, dialogue is in the center channel and effects are in the side channels. If the mixing ratios aren't right, the dialogue can end up in the background — that's why speech can be hard to hear when you extract audio from a film.

Some tools offer "dialogue-focused" options that mix the center channel more strongly.

PCM: the lossless source

In some videos the audio is stored completely uncompressed, in PCM format. It's found in professional cameras, some screen recording software and Blu-ray discs.

PCM is the same data as what's inside WAV files. In that case:

• Extracting to WAV: a direct copy, no loss at all.
• Extracting to FLAC: lossless compression, the data is preserved exactly, the file shrinks.
• Extracting to MP3/AAC: the first lossy encoding; the quality depends entirely on the bitrate you choose and at a good bitrate the result is excellent.

Extractions from PCM sources are the best starting point for editing.

In summary

In a video file, audio sits as a separate track packaged interleaved with the video data; the first step of extraction is separating those packets (demuxing) and that step involves no decoding. If the target format is compatible with the source codec, the packets are carried as they are — a stream copy, zero loss, an operation taking seconds. If not, decoding and re-encoding are required and a second loss is added on lossy sources. Timestamps determine sync and are the main cause of drift with variable frame rate sources. Downmixing multi-channel film audio to stereo can weaken dialogue — a natural consequence of the mixing ratios.