How to seek a specific frame quickly?

[Akascape]

I want to get a particular video frame using pyav. I found that container.seek method only seeks to time(int) offset, so it will not help in finding the required frame.

Perhaps, I used this method to get the required frame:

container.seek(0, whence='time', backward=True) # seek to the starting
frame_num = 30  # the frame I want to seek
time.sleep(0.2) # limit cpu usage
for _ in range(frame_num): 
    frame = next(container.decode(video=0))
return frame

This method works properly but it is too slow while seeking the last frames of any long video file. (with frames>1000)

I tried to solve this issue by modifying the seek method :

frame_num = 30
framerate = container.streams.video[0].average_rate # get the frame rate
sec = int(frame_num/framerate) # timestamp (sec) for that frame_num
container.seek(sec*1000000, whence='time', backward=True)  # seek to that nearest timestamp
sec_frame = sec * framerate # get the key frame number of that timestamp
       
for _ in range(sec_frame, frame_num):
    frame = next(container.decode(video=0))
return frame

This reduces the time while seeking any frame of the video, but it doesn't return the proper frame because container.seek method doesn't seek to the proper timestamp. I found that container.seek is giving the same key frame for consecutive 3 seconds. I don't know if I am missing something...

Is there any other way to get the required frame with pyav?

[Akascape]

So, I somehow managed to fix the lag issue:

frame_num = 1000 # the frame I want
framerate = container.streams.video[0].average_rate # get the frame rate
time_base = container.streams.video[0].time_base # get the time base
sec = int(frame_num/framerate) # timestamp for that frame_num
container.seek(sec*1000000, whence='time', backward=True)  # seek to that nearest timestamp
frame = next(container.decode(video=0)) # get the next available frame
sec_frame = int(frame.pts * time_base * framerate) # get the proper key frame number of that timestamp

for _ in range(sec_frame, frame_num):
    frame = next(container.decode(video=0))
return frame

Now it returns the required frame in less than a second.

[Akascape]

@Laleee Yes, this method is seeking the exact frame without any delay.
Here is a video player I made using pyav in which I implemented this method for accurate frame seeking:

out.mp4

I actually modified tkvideoplayer, (a module to display video files in tkinter UI using pyav)
But I noticed that the cpu usage increases if you keep on seeking frames continuously, it can be minimized by adding a time delay.

[Jafner]

Can you share what the container and codec timebases were for the test file you were using?

I'm struggling to understand why multiplying the offset passed to .seek by a million would be useful agnostic of the video timebase.

I'm building a project similar to your tkVideoPlayer and your work has been invaluable in helping me learn some of the basics.

[Akascape]

@Jafner The conatainer is simply av.open(path)

the seek time is given in av.time_base, the multiplication is to correct the frame

You can see the full source code of the video player here: https://github.com/Akascape/tkVideoPlayer/blob/master/tkVideoPlayer%2Ftkvideoplayer.py

[Jafner]

Thanks for getting back so quickly. I'd been looking at that file, but wasn't able to deduce the reason for the multiplication.

Regardless, I ran the same code against my own test file and it worked exactly as expected after removing the whence='time' keyword argument from the .seek() method. It looks like that was deprecated in PyAV 7.0.0.

A simple print(av.time_base) returns 1000000, which explains why that multiplication corrects the timestamp for the frame.

I suppose it makes sense for av to have its own default time base (with a high number like 1,000,000 to minimize stream deformation). I just assumed the container's timebase would be preferred.

[Jafner]

I think I understand better now why we use the av.time_base, rather than container timebase.

1. The frame rate given by container.streams.video[0].average_rate is imprecise. For example, my video recorded at "60" FPS with a time_base of 1/1000 cannot place a frame exactly 60 times per second. It has to round to the nearest millisecond (e.g. 0, 17, 33, 50, 67, 83,....
2. The container.seek() method takes an integer representing an exact presentation timestamp (pts). So trying to calculate the pts from the frame index and framerate will fail whenever the container's time base cannot precisely represent the intended timing of the frame (i.e. ((frame_index * framerate) / time_base) % 1 != 0)
3. I was able to use round() to calculate the timestamp of the desired frame in the container's timebase with:

import av
with av.open("TestClips/x264Source.mkv") as container:
    frame_num = 622
    time_base = container.streams.video[0].time_base
    framerate = container.streams.video[0].average_rate
    frame_container_pts = round((frame_num / framerate) / time_base)
    
    container.seek(frame_container_pts, backward=True, stream=container.streams.video[0])
    frame = next(container.decode(video=0))
    return frame

Maybe this will help someone else who might prefer to use the container's timebase instead of av's.