Skip to content

Conversation

Quantizr
Copy link
Contributor

depends on #35595

Stores microphone audio in qcamera.ts as an AAC encoded stream at 32kbps

Audio can be added to any video output that supports audio streams by adding .include_audio to the encoder info in loggerd.h. Audio codec and bitrate is set in video_writer.cc.

@adeebshihadeh adeebshihadeh modified the milestones: 0.10, 0.9.10 Jun 30, 2025
@Quantizr
Copy link
Contributor Author

Quantizr commented Jul 3, 2025

moved audio encoder out of video writer to encoderd/encoders, but due to some limitations of libav I can't get it as clean as I originally imagined

I pushed the code I've been writing in this commit 56c0cfa (not pushed to the PR branch), it mostly works, I still need to clean it up and squash a few bugs

main thing is, it's rather complex to pass the required data between processes in a way that supports different audio codecs if we want to easily change codecs in the future
I can simplify things by only supporting AAC and hardcoding the required fields to be passed in the capnp struct, and always assuming AAC on both ends.
56c0cfa is currently in a weird in-between state in terms of switching codecs

Regardless of any of the above, the code in this PR right now would be significantly less complex and shorter than moving the audio encoder out, even with the simplified assumptions, and it is decently easy to change things like codec

My problem with the code in this PR it is that, previously, video_writer was only writing encoded video, with encoders being separate, and it felt weird doing audio encoding in video_writer instead of encoderd. However, I'm increasingly thinking that the complexity of moving the encoder out, resyncing the encoded audio (which is at a different rate and not even guaranteed to be constant) with the video, and also handling the interpretation of the codec on both sides isn't worth it.

I will think about this more when I wake up

@Quantizr
Copy link
Contributor Author

Quantizr commented Jul 3, 2025

fixed and cleaned up the original implementation which encodes and writes audio in video_writer.cc

@Quantizr Quantizr requested a review from adeebshihadeh July 3, 2025 22:04
@Quantizr
Copy link
Contributor Author

Quantizr commented Jul 4, 2025

The program needs to receive rawAudioData to get the sample rate but can't write the corresponding audioData until VideoWriter is initialized and a header is written.

The program needs to receive the header of an encoded video messages in order to initialize VideoWriter and write a header, but can't write the header until it gets the sample rate from rawAudioData

@Quantizr
Copy link
Contributor Author

Quantizr commented Jul 4, 2025

Restructured the interactions between handle_encoder_msg() and write_encode_data() in loggerd such that the VideoWriter for each segment is created separately from the actual writing of encoded video data.

Then handle_encoder_msg() queues video messages until the rawAudioData message arrives with sample_rate, so that the audio contexts can be initialized.

Once the audio contexts are initialized, the queued encoded video frames are processed, with the header being written before with the first frame.

@Quantizr
Copy link
Contributor Author

Quantizr commented Jul 4, 2025

tldr: everything has to be set up before writing header, this includes audio sample_rate which we only have from rawAudioData

old version: video and audio init with hardcoded sample_rate -> video writes header and frame -> audio writes frame

new version: video init constructor and queue video frame -> audio init with sample_rate from rawAudioData and buffer frame -> video writes header and video writes queued frame -> audio writes queued frame

@Quantizr Quantizr requested a review from adeebshihadeh July 4, 2025 06:06
@adeebshihadeh adeebshihadeh merged commit a4e4a8a into master Jul 5, 2025
15 of 16 checks passed
@adeebshihadeh adeebshihadeh deleted the audio-in-qcam branch July 5, 2025 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants