How Does Extracting Audio from Video Work? The Road from MP4 to MP3

What is inside a video file?

Looked at from the outside, an MP4 file appears to be a single whole. From the inside the situation is different: MP4 is a container, a box into which different parts have been placed in an orderly fashion.

A typical MP4 contains the following:

• Video track: the moving picture data
• Audio track: the sound itself
• Metadata: duration, resolution, frame rate, shooting information
• Sync information: which piece of audio lines up with which frame

The most important point here is this: audio and picture are stored separately. They are not mixed together or interleaved into one another. They are like two strips running side by side; the player reads both at once and aligns them by consulting the sync information.

For this reason, "extracting audio from video" is not really an act of creation but an act of separation. The audio is already there; what you are doing is taking it into a file of its own.

Why does the separation not finish instantly?

If the audio is already inside the file, why is it not simply copied?

In some cases it really can be copied. If the target format accepts the audio codec in the source, the audio track can be taken as it is and placed into a new container. That operation is fast and creates no quality loss.

But with MP3 as the target the situation is different. To understand why, we need to bring in a second concept: the codec.

The codec determines which algorithm was used to compress the audio. In MP4 files the audio is typically compressed with AAC. An .mp3 file, by definition, carries audio compressed with the MP3 codec. These are two different algorithms; the data one produces is not something the other can read.

So the process works like this:

1. The audio track is separated from the container
2. The AAC data is decoded and turned into a raw sound wave
3. That raw wave is handed to the MP3 encoder and compressed from scratch
4. The result is written out as an .mp3 file

The third step gives rise to the most important sentence in this article.

The honest point: there is quality loss here

AAC and MP3 are both lossy formats. Lossy compression works like this: the algorithm shrinks the file by discarding the information it judges the human ear least likely to notice. The discarded information is gone permanently and cannot be brought back.

When your video's audio track was compressed as AAC, some information was already discarded. Now, when you decode that audio and hand it to the MP3 encoder, a second algorithm performs another round of filtering according to its own rules.

When two different filterings stack on top of one another, the result is somewhat more degraded than audio that has been through a single one. This is a technical fact and it should not be hidden.

But how much does it matter? Here too one has to be honest: in practice, in most cases, very little. At a reasonable bitrate — say 192 kbps and above for music — it is very hard to tell the difference under normal listening conditions. In recordings that contain speech only, the loss is barely felt at all.

The places where the loss becomes obvious are well known: high-frequency and complex sounds. Cymbals, whistling, dense orchestration, rapid transitions. A metallic edge or a dullness can appear in those spots. If you chose a low bitrate, those traces become far easier to hear.

The role of bitrate

Bitrate determines how many kilobits per second the MP3 encoder will use to represent the audio. The more generous you are, the less information the encoder is forced to discard.

Because you are compressing for a second time, being stingy here costs you more than usual. If you were producing MP3 from a fresh recording, a low bitrate would be more forgivable; but when you are filtering a signal that has already been filtered once, you need to leave the encoder some room to breathe.

A rough roadmap: 128 kbps is enough for speech only, 192 kbps is a reasonable lower bound for music and mixed content, and 256 kbps is a comfortable choice for music you are going to listen to attentively.

There is an upper bound too: you cannot exceed the quality of the source. If the AAC audio in the video is already at a low bitrate, pushing the MP3 very high will not bring back the lost detail, it will only make the file bigger.

Practical consequences

Once you understand this mechanism, three behaviours become correct of their own accord.

Keep the chain short. Every lossy conversion adds new loss and these accumulate. Go from the most original source you have to your target in a single step. When you extract MP3 from MP4, make your source the original video, not an intermediate file that has been converted before.

Keep the original. The MP3 output is a version that has lost information relative to the source. If you need a different format later, start from the original video rather than the MP3.

Choose your target according to your purpose. Are you extracting this audio to listen to it, or to work on it? For listening, MP3 is the right choice: small, portable, plays everywhere. But if you are going to edit the audio, cut it, apply effects or bring it into an editing program, WAV is a more suitable target, because being uncompressed it adds no new loss while you work on it.

Cases where the audio cannot be extracted

Knowing this mechanism also lets you understand why things happen when the process does not turn out as you expected. There are a few scenarios you may run into.

The video may have no audio track at all. Some screen recordings are captured with system audio off, some cameras shoot silently, and some edited videos have had the audio track removed entirely. In that case there is nothing to extract — a converter cannot produce a track that does not exist. When in doubt, open the MP4 in a player and confirm there really is audio.

The audio track may be corrupted. In a file that was half downloaded or cut short during recording, the audio track may be incomplete or faulty. In such cases the end of the output will be missing. Checking by playing the source through to the end catches this problem in advance.

The audio may be very quiet. If the source's own recording level is low, the output will be quiet too. There is an important distinction here: conversion does not raise the level, because its job is to carry the signal, not to change it. If you want to correct the level, that is an audio editing job, and for that job it is more sensible to work with an uncompressed source, meaning WAV.

The common thread is this: conversion does not produce what is not in the source. It does not bring back lost detail, it does not repair broken audio, and it does not raise a quiet recording. The only thing it does is carry the existing audio track into the format you want.

In summary

Extracting audio from video is not producing something that does not exist, it is separating out an audio track that is already sitting inside the container. That is why the operation is light — the picture is never touched. However, because the audio inside an MP4 is usually AAC and MP3 is a different codec, this separation also involves a re-encode. That move between two lossy formats creates a small but real quality loss. By choosing a reasonable bitrate you can make that loss practically inaudible; what you should not do is walk the same audio repeatedly between different lossy formats.