Stream Copy vs. Re-encoding in Video Merging: The Technical Difference

Merging two videos sounds simple: put one after the other. In reality, codec configurations, keyframe boundaries and timestamp arithmetic come into play behind the scenes — and they determine whether the operation takes seconds or minutes and whether the quality is preserved.

Two basic approaches

Stream copy (concat demuxer): the video and audio data are carried into the new file without being decoded at all. Only the container is rewritten and the timestamps are corrected.

• Quality loss: zero
• Time: file copy speed
• Condition: strict parameter compatibility

Re-encoding (transcode / concat filter): all the pieces are decoded, converted to a common format, and re-encoded as a single stream.

• Quality loss: yes
• Time: minutes or hours
• Condition: none, always possible

Why stream-copying demands such strict conditions

Before a video decoder starts decoding a stream, it receives configuration information. In H.264 that's called the SPS (Sequence Parameter Set) and PPS (Picture Parameter Set) and it contains:

• Resolution (width, height)
• Profile and level (Baseline/Main/High)
• Chroma format (4:2:0, 4:2:2, 4:4:4)
• Bit depth
• Frame reordering information

That information is given at the start of the stream and the decoder sets itself up accordingly — allocating memory, determining buffer sizes.

Now imagine merging a 1080p video with a 720p video by stream-copying. The resolution changes mid-stream and the decoder has to be reinitialized. Some players can do that, most can't — the result is corrupted picture, freezing, or the player crashing.

That's why the compatibility list for stream-copying is:

| Parameter | Must it match | |---|---| | Video codec | Yes, absolutely | | Resolution | Yes, absolutely | | Profile / level | Yes | | Chroma format | Yes | | Frame rate | Yes (for timing) | | Audio codec | Yes | | Audio sample rate | Yes | | Audio channel count | Yes | | Bitrate | No (can vary) |

That last row matters: bitrate already varies within a stream, so it doesn't need to match.

MKV's flexibility

The MP4 container expects a single configuration at track level. Changing resolution or profile mid-stream is non-standard.

MKV is more flexible: it has a structure that can carry sections with different parameters in the same file, and many MKV players adapt to mid-stream changes.

The practical consequence: merging mixed sources in MKV is sometimes possible without re-encoding, while it isn't in MP4. But that's player-dependent behavior and not guaranteed. If you're distributing the file to a wide audience, the reliable route is re-encoding.

Rewriting the timestamps

Even when stream-copying, one thing definitely changes: the timestamps.

Every video file starts its timestamps from zero. The first video is stamped from 0 to 120 seconds, the second again from 0 to 90 seconds.

If you simply append them, the player sees the second video's stamps and thinks it's going back to the beginning. The result: the video ends at second 120 or the player gets confused.

The merging tool adds the first video's duration to all of the second video's timestamps:

Video 1: PTS 0, 40, 80, ... 119960 (ms)
Video 2 (original): PTS 0, 40, 80, ... 89960
Video 2 (shifted): PTS 120000, 120040, ... 209960

That creates a continuous timeline and the player sees the video as a single stream.

The same operation is done for the audio track, and shifting audio and video by the same amount is critical for preserving sync.

The keyframe boundary

Video codecs store frames in three types:

• I-frame (keyframe): a complete picture on its own. Can be decoded independently.
• P-frame: depends on previous frames.
• B-frame: depends on both previous and following frames.

These are arranged in GOP (Group of Pictures) groups, and every GOP starts with an I-frame.

Its effect on merging: when stream-copying, the cut point has to land on an I-frame. You can't start from a P-frame because that frame requires the frames before it, and those aren't in the new file.

The consequence: if you're cutting and merging video using stream copy, the tool rounds the cut point to the nearest keyframe. It can end up cutting a few seconds before or after the second you wanted.

If you want frame-accurate cutting, that section has to be re-encoded. Some tools use a smart approach: they re-encode only the GOPs at the cut points and stream-copy the rest. That's known as "smart rendering" and gives both speed and precision.

If you're merging complete files (with no cutting) you don't hit this problem — every file already starts with an I-frame.

The variable frame rate problem

Constant frame rate (CFR): frames arrive at equal intervals. The timestamps are a regular arithmetic sequence.

Variable frame rate (VFR): frames don't arrive at equal intervals. Phone cameras drop frames depending on light, screen recording software depending on the absence of motion.

VFR creates two problems in merging:

1. The duration calculation comes out wrong. If the tool assumes an average frame rate and computes duration as frame count / fps, it gets a value different from the real duration. The second video's shift amount is computed wrongly.

2. The drift accumulates. If there's a small error in every piece, they add up. When you merge five pieces, a noticeable lip sync problem appears toward the end.

The symptom is characteristic: sync is fine at the start of the video and breaks toward the end.

The fix: converting the pieces to a constant frame rate before merging. That's an extra re-encoding step but it solves the problem at the root.

If you regularly work with phone-shot video, making that step part of your workflow is a good habit.

Compatibility on the audio side

Similar conditions apply to audio:

Codec compatibility: if one is AAC and the other MP3, stream-copying isn't possible.

Sample rate: a mix of 44.1 kHz and 48 kHz, if not converted properly, causes one piece to play at the wrong speed.

Channel count: a mix of mono and stereo requires conversion.

Also, because lossy audio codecs work on blocks, every file has encoder delay and padding samples at its start and end. When you merge by stream-copying, those gaps are preserved and a short silence appears at every transition.

There's no such problem on the video side (video frames need no block padding), so a small timing difference can form between audio and video. Across long merges that can accumulate.

If you want gapless audio, the audio has to be re-encoded.

Choosing targets when re-encoding

If re-encoding is unavoidable, you have to choose the target parameters:

Resolution. Target the most commonly used value. If the mix has four 1080p pieces and one 720p, choose 1080p — only one piece gets upscaled.

Frame rate. Choosing the highest gives a smoother result. Raising a 30 fps video to 60 fps repeats frames (it adds no new information) but the 60 fps pieces keep their smoothness.

Aspect ratio. If there are different ratios, adding black bars (letterbox/pillarbox) preserves the content; cropping fills the screen but loses content.

Bitrate / quality. Using a CRF-based quality target usually gives a better result than constant bitrate, because every scene gets as much data as it needs.

Codec. H.264 is the safest choice; it works everywhere.

In summary

In video merging, stream-copying is the ideal route — no quality loss and done in seconds — but it has strict conditions: the codec, resolution, profile, chroma format, frame rate and all the audio parameters have to match. Even when that condition holds, the timestamps have to be rewritten; shifting audio and video by the same amount determines sync. If you're cutting, stream-copying runs into keyframe boundaries and the cut point gets rounded. Variable frame rate sources are the most common cause of cumulative sync drift and should be converted to a constant frame rate in advance. If compatibility can't be achieved, re-encoding is required and the target resolution, frame rate and aspect ratio decisions should be made deliberately.