FPS aquired vs fps written for play back

[aaronjnorris]

We have had difficulty using general USB cameras for recording videos and then having the length of the recorded video match the time length of the recording because the frame rate set can be set within the video capture node doesn't match the rate actually achieved. We made the script here for troubleshooting this problem. I have seen other discussions on the topic but have not seen any solutions that might work more generally. I would like to have the video writer write at the speed the frames are obtained. Any suggestions on how to do this?
We have been using ELP and Logitech cameras to avoid costs and allow for scaling. We do not need particularly high frame rates (30-60 would be enough). The ability to control these cameras via Bonsai varies and the set frame rate is often not attained in the recording.
I have seen that the interframe interval and be quite variable presumably due to computer delays. Any suggestions on how to control the frame rates of these cameras other than options in the video writer node that appear?

Would also love suggestions on the linked script that adds frame rate, number, and interval to the video!

https://1drv.ms/u/s!AsluozMIo0dagfF3Mx1BAtilJsyLvA?e=KyZRgh

[bruno-f-cruz]

Have you tried to adjust the exposure (make sure it is not doing automatic exposure)

[aaronjnorris]

yes - helps maintain consistency but not always getting frames at the rate set in Bonsai

[jbscherrer]

The framerate of the camera device is set by an internal clock in the camera itself and will be independent of camera-computer communication timing. The interframe interval variation that you see in the script is due to latency in USB communication and CPU timing, or as Bruno has said, potentially automatic exposure on the camera chip.

[aaronjnorris]

Thanks for the input and help. Is there a way to get Bonsai to record video for playback at the rate frames arrive? If the rate is slightly less than set in the video writer (ex actual 28fsp vs set in writer at 30 fps) then the videos end up short.

[bruno-f-cruz]

You can change the framerate parameter in videowriter to 28.

…

On Tue, 8 Nov 2022 at 23:44, aaronjnorris ***@***.***> wrote: Thanks for the input and help. Is there a way to get Bonsai to record video for playback at the rate frames arrive? If the rate is slightly less than set in the video writer (ex actual 28fsp vs set in writer at 30 fps) then the videos end up short. — Reply to this email directly, view it on GitHub <#1112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVZBB32QJX4T5SSM2K7VCLWHLQXBANCNFSM6AAAAAARWTXRXM> . You are receiving this because you commented.Message ID: ***@***.***>

[chongtianyifa]

I know the discrepancy concerning the duration is caused by the difference between the rate of camera capture and that of encoding by software. But we are talking about the relationship between the number of frames and the real recording duration.

1. Should be my bonsai test with timer your 'alternative way' as fps is defined at 30fps and duration at 2min? However, the number of frames is smaller than 30 x 60 x 2.

2. Also, if 'your camera is likely acquiring frames at a slightly higher speed than 30fps', then more number of frames encoded at 30 fps will result in a video longer than 2 minutes rather shorter.

3. Importantly, my test is exactly what you said. The real duration is controlled by observing my timer of cellphone. The longer the real recording duration is, bonsai dropped more frames while OBS always gave a video with exact length and frame numbers.

[bruno-f-cruz]

Those 60x2 seconds, for bonsai, determine the duration the camera is acquired for, not the duration of the output encoded video. If you keep everything the same but change the encoding fps for 15fps, the encoded video will have double the duration. I guess what I am trying to say is that the duration of the output video is a bit meaningless for the most part. As jbscherrer stated, what we usually care about is "when" does that data correspond to in "real-time" units.
Regardless, If you really want an output video of exactly 1min, saved at 30fps, use the Take operator instead of the timer, and set the Take property to fps*seconds. However, this approach rests on the assumption that your camera is acquiring at exactly the encoded fps of the output video which might not be the case as we discussed before.

PS: you are absolutely correct, I should have written less than 30fps, not more. Sorry!

[chongtianyifa]

Yes, I agree with that the duration of the output video is not important as it was just determined by the number of frames/the encoding fps defined at metadata. But I still don't get why OBS can save the exact number of frames matching the encoding fps*real time while Bonsai saved a smaller amount.

[bruno-f-cruz]

Not sure. My best guess is what I pointed out before. OBS tries to save a video with the duration you specified, even if that means recording more, or less, data from "real world time".

[aaronjnorris]

In my experience, I found "short" videos from Bonsai due to an average frame rate acquisition that is below the fps set in video writer. All of the captured frames are present - they are played back more quickly than they were recorded - in other words at a higher frame rate. I wonder how standard video recording programs are able to correct for slower or changing incoming frame rates (ex shift to a darker scene triggering lower fps to lengthen exposure) to set the playback rate and if it is possible to do something similar in bonsai.

[IamJuanHilario]

@aaronjnorris Did you find a solution to this problem?, I am currently facing the same issue, but it extends to also not aquiring csv writer files to the specifed precision of for example 1 minute when I use a "timer" connected to a "workflow out" module to end the experiment

[aaronjnorris]

No. I continue to struggle with how Bonsai interacts with USB cameras. I am sure I do not understand some fundamental concepts about how USB cameras are controlled.
For some cameras, the amcap program here https://www.svpro.cc/download/ can help with setting settings that will be used by Bonsai despite not being accessible in menus in Bonsai. Making sure the exposure is set to manual is helpful in maintaining consistency. I don't know enough to understand how different programs access the USB camera settings or how the settings are negotiated between the camera and the program. in some cases settings available amcap, for example, can not be set in bonsai.

For many parameters that we examine time between frames is important to know. I have taken to checking that the frame rate and interframe interval are what I think it is - often it is not what is set - I am guessing due to hardware limitations???
Here is a version of a script I use to add this information to the video directly and to check before recording what the
Video_FPSCheck.zip
I try to make sure the actual frame rate, interframe interval, and the set encoding rate all jive.

If anyone has better options or insight into how to better control the frame rate, exposure, gain, etc of USB cameras from in Bonsai I would greatly appreciate the information.

[bruno-f-cruz]

I am not sure your question is related to this post. Can you open a new discussion post with the relevant example? Thanks.

Regarding the original post, I am still not sure what the actual question is. Bonsai will let you set a target fps exposure, etc... in the camera. Once that's done bonsai will just listen to incoming frames, it will not attempt to interfere with the data stream in any way. The same happens with videowriter, whatever frames the camera issues, the writer will encode and save to the file. It will not interpolate frames, it will not duplicate frames (something that obs does depending on your encoding settings). What you see is what you get (or the other way around in this case I guess haha)