-
Notifications
You must be signed in to change notification settings - Fork 10.1k
include audio in qcamera.ts #35608
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
include audio in qcamera.ts #35608
Conversation
…er QAvg/info messages
…mance for defaults
moved audio encoder out of video writer to encoderd/encoders, but due to some limitations of libav I can't get it as clean as I originally imagined I pushed the code I've been writing in this commit 56c0cfa (not pushed to the PR branch), it mostly works, I still need to clean it up and squash a few bugs main thing is, it's rather complex to pass the required data between processes in a way that supports different audio codecs if we want to easily change codecs in the future Regardless of any of the above, the code in this PR right now would be significantly less complex and shorter than moving the audio encoder out, even with the simplified assumptions, and it is decently easy to change things like codec My problem with the code in this PR it is that, previously, video_writer was only writing encoded video, with encoders being separate, and it felt weird doing audio encoding in video_writer instead of encoderd. However, I'm increasingly thinking that the complexity of moving the encoder out, resyncing the encoded audio (which is at a different rate and not even guaranteed to be constant) with the video, and also handling the interpretation of the codec on both sides isn't worth it. I will think about this more when I wake up |
fixed and cleaned up the original implementation which encodes and writes audio in video_writer.cc |
The program needs to receive rawAudioData to get the sample rate but can't write the corresponding audioData until VideoWriter is initialized and a header is written. The program needs to receive the header of an encoded video messages in order to initialize VideoWriter and write a header, but can't write the header until it gets the sample rate from rawAudioData |
Restructured the interactions between Then Once the audio contexts are initialized, the queued encoded video frames are processed, with the header being written before with the first frame. |
tldr: everything has to be set up before writing header, this includes audio sample_rate which we only have from rawAudioData old version: video and audio init with hardcoded sample_rate -> video writes header and frame -> audio writes frame new version: video init constructor and queue video frame -> audio init with sample_rate from rawAudioData and buffer frame -> video writes header and video writes queued frame -> audio writes queued frame |
depends on #35595
Stores microphone audio in qcamera.ts as an AAC encoded stream at 32kbps
Audio can be added to any video output that supports audio streams by adding
.include_audio
to the encoder info in loggerd.h. Audio codec and bitrate is set in video_writer.cc.