Seeking to a frame by selectively loading av.Packets

[Michael-Equi]

Hello, I am trying to setup a data pipeline that requires H264 seeking over a special file format that retrieves only the bytes corresponding to a specific frame. I am able to retrieve the bytes corresponding to keyframes but it seems that that is not enough to perform decoding. When I start from the first frame and decode everything works fine but I cannot start from a random keyframe. I added some code the outlines what I am doing along with the error that is thrown when I try to read a random keyframe. I believe the issue is related to the codec parameters not being set correctly. Is there an interface that allows me to manually set these parameters? Alternatively, is. there a better way to accomplish this?

# Create an output container
codec = av.CodecContext.create('h264', 'r')

# Fetch a random keyframe from  a file that contains a lists of bytes 
# Each list of bytes corresponds to the encoded data from a single frame 
# A list of keyframes is maintained so we can selectively read a single keyframe without loading any additional data into memory
with open("frames.mcap", "rb") as f:
   msg = get_random_keyframe(f) # grabs a keyframe
   packet = av.Packet(msg.data)
   video_frames = codec.decode(packet) # Fails here if not the first frame
   video_frames[0].to_image().save(f'frames/frame_{i}.png')

Error:

Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
 (repeated 52293 more times)
non-existing PPS 0 referenced
decode_slice_header error

[skeskinen]

What is rb in with open("frames.mcap", "rb") as f: ?

It seems like the codec extradata is in the beginning of the stream. That would explain why starting from packet 0 works. I think this implies that codec_context.global_header = False.

You need to somehow get the correct extradata to your new codec. Either from the stream in the input container created by the library or manually from the beginning of the stream since it seems to be there.

Why not use the codec in the input container that the library created?