New experimental "Resonate protocol" for streaming synchronized multi-room audio and HiFi music playback to multiple media player appliances and smart speakers running ESPHome on ESP32 or Linux on Raspberry Pi (Zero)

[Hedda]

Not sure if this will be usable on all ESPHome based audio output/input devices running on ESP32 (i.e. voice assistants, media player, and smart speaker hardware), but FYI, I heard it sounds like (pun intended) some Open Home Foundation and Music Assistant developers have started working on a new experimental open-source Hi-Fi audio streaming protocol for a better multi-room audio and music playback experience on embedded hardware running ESPHome-based client on ESP32 as well as Linux-based client on the roadmap for Raspberry Pi (Zero):

• https://github.com/Resonate-Protocol (Resonate Protocol organization on GitHub including contact information for collaboration)
  • https://github.com/Resonate-Protocol/spec (WIP specification of the new experimental Resonate multi-room music experience protocol)
  • https://github.com/Resonate-Protocol/aioresonate (Async Python library implementing the Resonate Protocol.)
  • https://github.com/Resonate-Protocol/audio-sdk-js (Proof of concept of how Improv Audio can work in TypeScipt/JavaScript)

This new protocol specification is design from scratch but is similar to Squeezelite and Snapcast as well as Music Player Daemon (MPD) in concept at a high level, Resonate is not a stand-alone player, but an extension that can turn an existing audio player into a Sonos-like multiroom audio solution with time synchronized between the clients and the server to play perfectly synced audio. In their own tests they claim to see that average time deviation is below 0.05ms for synced audio (50 microseconds between two ESP32-S3 devices running ESPHome and connected over WiFi). Think of it like a completly royalty-free and open-source competition to Apple AirPlay (formerly AirTunes) and Google Cast (Chromecast Audio) propriatory protocols but for implementing HiFi multi-room audio solutions.

“Resonate is a multi-room music experience protocol. The goal of the protocol is to orchestrate all devices that make up the music listening experience. This includes outputting audio on multiple speakers simultaneously, screens and lights visualizing the audio or album art, and wall tablets providing media controls.” “Definitions; Server, a Resonate server. Orchestrates all devices. Generates an audio stream, manages all the players, provides metadata etc… Player, a Resonate client that can play audio, visualize audio or album art or provide music controls”.

Music Assistant lead developer posted a reply in their discussion section answering the question why they choose to not just use Snapcast:

• https://github.com/orgs/music-assistant/discussions/3883

With this new "Resonate" protocol the Music Assistant client/server and ESPHome will also get a new experimental audio player component that adds support for the this "Resonate Protocol" for synchronized music-playback across multiple devices, (i.e. audio synchronizer between ESPHome-based to enable sound-sync timing for DIY multi-room audio players), as well as ability to also switch codecs on-the-fly.

For a WIP reference implementation of a server using their aioresonate (Async Python library see how it is implemented in Music Assistant:

• https://github.com/music-assistant/server/tree/resonate/music_assistant/providers/resonate

⚠️ Note! The work-in-progress audio client component for the ESPHome firmware is still very much experimental and in early proof-of-concept phase, not near ready for production use, and the availabe code made available is just intended for testing or feedback purposes only. ⚠️

• WIP: Resonate synchronized audio component esphome/esphome#10183

Implementation Details:

• Time Synchronization:

  • Employs a Kalman filter to model and compensate for internal clock drift between the client and server
  • Dynamically tracks and adjusts for each device's clock characteristics to maintain tight synchronization
  • In preliminary testing, achieved a median audio synchronization error of approximately 50 microseconds between two ESP32-S3 devices connected over WiFi.

• Architecture:

  • The resonate component implements a flexible hub architecture that allows devices to participate in the audio listening experience with different levels of functionality:
    • Text Sensor: Display track metadata without audio playback (e.g., on screens)
    • Media Player: Output synchronized audio
    • Extensible Design: Ready for future integrations like audio visualization

• mDNS:

  • Implements mDNS advertisement for the component's WebSocket server
  • Enables automatic discovery by resonate servers

• External Dependencies:

  • Integrates esp-libopus (WIP IDF component) for Opus audio codec support

• Testing:

  • Currently compatible with:

    • Music Assistant Server - feat/improv-audio branch
      • Note: This implementation has known issues but provides basic testing capabilities for synchronized playback

  • Example entry for config.yaml:

    • Example Voice Preview Edition firmware is available here: https://github.com/esphome/home-assistant-voice-pe/blob/resonate-dev-snapshot1/home-assistant-voice.yaml (requires building with ESPHome dev branch)

• Current Status:

  • August 28-29th updates adds several important changes:
    • Now compatible with the Music Assistant server's resonate branch
      • https://github.com/music-assistant/server/tree/resonate/music_assistant/providers/resonate
    • Fixes issues with rapid stopping and starting (requires using the mixer and resampler speaker platform changes in this PR)
    • Improves memory handling and safety
    • Adds groundwork to support cover art as an image component in ESPHome
      • This PR is intended for testing and feedback purposes only. The implementation contains:
        • Numerous TODO items for simplification and optimization
        • Extensive debug logging for diagnosing synchronization issues
        • Incomplete features pending server-side support (metadata processing, FLAC decoding)
        • Breaking changes expected in future iterations
        • Modified Components

• Future Work:

  • Protocol refinement and stabilization
  • Enhanced audio stack to support additional synchronized audio protocols
  • Complete metadata and FLAC support
  • Production-ready implementation

There was some discussions + further explainations on its concept in a other thead that asked about this new multi-room streaming protocol:

• https://github.com/orgs/music-assistant/discussions/3883

Originally posted by @marcelveldt in #3883 (comment)

"we decided to create (yes, yet another!) protocol. One that would fulfill all our needs and would work across the internet and suitable for low powered devices like the ESP32. It will support (perfect) synced playback by default but can also switch codecs on the fly, making it suitable to stream to a (remote) browser. Based on websockets ensures it can travel across firewalls, reverse proxies and whatnot. It's a bit of shame to not use the strength of an existing audio streaming protocol (snapcast) but we ran into so much trouble it became more of a burden than a help. I even considered rewriting Snapcast from scratch in python, so keep the binary protocol the same but just re-implement it with our issues fixed. I'll make sure we publish the details somewhere soon, currently its still in ideation/POC phase but looking very good. "Be aware that this is nowhere near ready to test something - we are still defining the spec and doing some PoC implementations to test the theory."

If I understand correctly this new Resonate synchronized audio component will add a flexible hub architecture for audio time synchronization

• https://github.com/Resonate-Protocol/spec

This could solve the problem that you normally need more expensive hardware solution if you want to add multi-room audio to existing speakers, as once this is implemented fully you could have almost perfect multi-room audio syncronization using something like the Home Assistant Voice Preview Edition connecte to external speakers using its stereo output jack:

• https://www.home-assistant.io/voice-pe/

But a more relevant follow-up feature request for Music Assistant when this implemented would be if could add support for different types of multi-room audio, (to have options for single source-single zone, single source-multiple zone, or multiple source-multiple zone playback)?

• https://en.wikipedia.org/wiki/Multi-room_audio

Ongoing discussion about this new experimental Resonate audio component is going on in this Discord channel here:

• https://discord.com/channels/753947050995089438/1369735163629801532

KevinA:
"Luckily math is here to save the day 😆 . A Kalman filter is a nice way to combine measurements with different uncertainties, which is exactly what we have! Basically, if the most recent message has a very low delay, then we weigh its value much more than if it had a very high delay. It gives us a much more stable offset than using the median, but it also gives us an estimated error after each update. This means the client can estimate how accurate its own offset is! This would allow us to slow down or increase the pace we send the time messages. The more confident we are in the offset, the less we need to send - and vice versa. This will be especially helpful when we first connect and need to spam a bunch of time messages to quickly build our confidence in the computed offset.

So being accurate is important, of course, but being stable is also very important. The median was problematic, even if its inaccuracy wasn't audible. With the median filter, the offset would shift by over 200 microseconds, on average, per update, which means the client would have to constantly be tweaking the audio to try to stay in sync. With very big jumps, it would require hard syncing which is audible.

The Kalman filter modeling the drift (of 2 seconds per day) had an average offset change of only ~25 microseconds per update. Keep in mind, if the clock does drift 2 seconds per day, we would expect it change 23 microseconds per update if the updates are 1 second apart. If there is no systematic drift in the client clock, the Kalman filter had an average change of less than 1 microsecond per update."

Btw, looking at the commit history it seems like "Resonate" used to previously be refered to as "improv-audio" internally by its developers:

• music-assistant/server@dev...feat/improv-audio

PS: Somewhat ironically the Open Home Foundation recenrly posted a newsletter blog post titiled "" and then they choose to use xkcd's "standards" classic XKCD comic strip on how standards proliferate as their tumbnail for the Resonate organization on GitHub, however with the Open Home Foundation, Home Assistant , Music Assistant, and Nabu Casa founder backing this they probably have the influence to at least convince many open-source based media layer implementations to also add support for this new protocol:

• https://newsletter.openhomefoundation.org/why-isnt-there-one-standard/

[trisweb]

A few questions come to mind:

What is the audio quality of this new protocol?

Does it support high resolution audio?

Does it use lossless encoding?

Does it support clients other than ESP32 based microcontrollers?

[OzGav]

The protocol doesn't limit the quality.
Yes
Can do
Yes

[Hedda]

@trisweb It is also important to remember that even if the protocol specification supports higher quality or more features then each client implementations can still have suffer from poor implementation of required standards. Meaning it will probably primarly be hardware devices/appliances that will limit the top performance. Like example the Home Assistant Voice Preview Edition hardware board which just uses a TI AIC3204 (TLV320AIC3204 audio codec and DAC) for its 3.5 mm stereo headphone audio output jack which is relativly low-end and limits it to 48 kHz sampling rate output. So to get higher sampling rates outputs then you either need to use different hardware with a better DAC if you want analogue output directly or find a solution that provide digital output compatible with other encodings so that you can use an external audio decoder (audio codec) and DAC (Digital to Analog converter). The great thing is that not only is this new protocol specification is completly royalty-free and open-source but the Open Home Foundation has also created an open standard for the hardware too (so the Home Assistant Voice PE hardware is more or less just a proof-of-concept and development platform to build upon), which means anyone can make compatible hardware that is much better:

• https://www.home-assistant.io/blog/2024/12/19/voice-preview-edition-the-era-of-open-voice/

Anyway, while I do not know more than what is written in the current specification of the Resontate protocol (which is a living work-in-progress with nothing set in stone), I would still assume we can make an educated guess that their long-term plan is for the specifications in this new protocol/standard to either be better or at least have feature-parity with all the "competing" player provider protocols, meaning it should in the future best or match the features and functions of all the other player providers listed in the player capabilities comparison table here (where it right now on paper looks like Squeezelite is the one they need to best today in the case of pure audio quality, though upstream Snapcast and MPD also support higher audio quality and more features):

• https://www.music-assistant.io/player-support/

[Hedda]

@marcelveldt do you also plan to make a default/reference "Resonate Media Player" client for Linux-based running on Raspberry Pi hardware?

Perhaps even a complete "Hi-Fi" media player client Linux distribution (based HAOS/HassOS if not LibreELEC OS or similar JeOS Linux distro)?

That is, as there is no guarantee that third-party media players will adopt/implement support for this Resonate protocol I was wondering it the Open Home Foundation plan to make a reference media player software application for Linux that support Resonate protocol on Raspberry Pi?

Asking since Raspberry Pi running Linux is today probably the most popular platform for DIY line-out WiFi client solution for multi-room audio.

Suggesting a stand-alone all-in-one music player component package for Linux to make it simple to setup a Resonate client media player and achieve audio synchronized playback that more or less matches what kahrendt is developing for ESPHome here, (but for Raspberry Pi instead):

• WIP: Resonate synchronized audio component esphome/esphome#10183

Example a Raspberry Pi with either HiFi DAC + Amplifier hardware HATs or a USB Audio Device Interface adapter for audio output like these:

• Sonocotta Raspberry PI Media Center (open source hardware Hi-Fi DAC + Amp HATs for Raspberry Pi from @sonocotta / @anabolyc)
  • https://github.com/sonocotta/raspberry-media-center
• HifiBerry (probably at this time the most popular DAC/AMP HAT products for the Raspberry Pi)
• Pimoroni Pirate Audio (series of HATs for Raspberry Pi)
• Raspiaudio Pi Audio Drive (Raspberry Pi 5 expansion board with HD DAC, audio I/O and AMP)
• IQaudio DAC HAT (no longer being manufactured?)
• JustBoom DAC HAT (no longer being manufactured?)
• Volumio (commercial music network streamer OS based on MPD that run on Raspberry Pi which can also be used on DIY hardware)
• RuneAudio (abandoned open-source music streamer with "RuneOS" disto based on MPD player that was designed to run on Raspberry Pi)
• Behringer USB Audio Device Interface adapters (ex. their UCA222 and UCA202 which has analog line-out + optical S/PDIF out + line-in)
  • For reference, this type of SB Audio Device Interface adapter will probably be the easiest for developers and testers to get started with.

PS: FYI, indirectly related is that synesthesiam begun working on a Linux-based Voice Satellite for Home Assistant using the ESPHome protocol (i.e. porting ESPHome to Linux) as the successor/replacement for Rhasspy's Wyoming Satellite remote voice satellites on Raspberry Pi hardware:

• https://www.reddit.com/r/homeassistant/comments/1na1chk/new_wyoming_satellite_successor/

  • https://github.com/OHF-Voice/linux-voice-assistant

Quoting synesthesiam comment/reply on that Reddit:

Author here, glad to see so much interest :)

To give some context: the Wyoming protocol was developed for voice services like speech-to-text, text-to-speech, etc. I added satellite messages early on so we could use Raspberry Pi's alongside the ESPHome devices. Over time, the Wyoming Satellite feature set has fallen behind ESPHome and it's felt like I'm basically recreating the ESPHome messages in Wyoming.

So I had the idea: why not just implement a tiny subset of the ESPHome protocol on Linux and reuse what's already there? This turned out to be way easier than I expected! It also has the benefit that I can expose the media player properly now, and it will be possible to expose LED rings and buttons too in the future without any changes to Home Assistant. The Raspberry Pi satellite just looks like an ESPHome device with the proper entities already in place.

At the moment, this project only supports microWakeWord but I'd like to add openWakeWord support soon (PRs welcome). I'm also using the mpv media player directly now for audio output so I can mix audio streams properly. Things like timers, announcements, start conversation, audio ducking, etc. all just work out of the box. It should also be possible to use PipeWire/PulseAudio plugins for acoustic echo cancellation (AEC) so the wake word can still be heard over music.

The next step will be to add a proper tutorial and start collecting feedback from beta users :)

[marcelveldt]

Yeah, @maximmaxim345 is going to implement a full functional client into the aioresonate library

[Hedda]

@maximmaxim345 any plan to implement the aioresonate library into the Linux voice assistant project that synesthesiam is working on?

• https://github.com/OHF-Voice/linux-voice-assistant

[trisweb]

General +100 to a Linux client. Ideally a fully functioning service one can run in place of squeezelite.

[maximmaxim345]

@maximmaxim345 any plan to implement the aioresonate library into the Linux voice assistant project that synesthesiam is working on?

No plans yet, but shouldn't be too difficult to add once the linux client is ready.

[Hedda]

@maximmaxim345 have you by the way looked at Matter Casting APIs & architecture to see if it has any good concepts that can be borrowed?

• https://github.com/project-chip/connectedhomeip/tree/master/examples/tv-casting-app

  • • https://github.com/project-chip/connectedhomeip/blob/master/examples/tv-casting-app/APIs.md

• https://github.com/project-chip/connectedhomeip/tree/master/examples/tv-app

  • https://github.com/project-chip/connectedhomeip/blob/master/examples/tv-app/linux/README.md

PS: One major limitation is the the current Matter Casting specification is that they are still missing support for the audio-only part of its architecture as their casting technology focus so far has just been on video streaming and this there is not yet any audio-only APIs or examples for smart-speakers or music-recievers, but I understand that some Connectivity Standards Alliance members have started to look at that now:

• [Feature] Matter Casting support for "Audio Player Architecture" with a new "Casting Audio Player device type" and "Audio Player endpoint" ("Casting Audio Player" and adding "Basic Audio Player") + look into adding multi-room music streaming for "Matter Speakers"? project-chip/connectedhomeip#31389