Resynthesis: Have AI generate a voice for MiniDexed that approximates a sample

[probonopd]

Traditionally, FM synthesizers have been (wrongly?) known as almost impossibly hard to program "realistically sounding" voices with.

Some sound designers have mastered this "black art", but for "the rest of us"... what if we could record some sound (like on a sampler), and let some AI magic create a DX7 patch matching it closely?

Some theory:
https://fcaspe.github.io/ddx7/

There is even source available to do that:
https://github.com/spiegelib/vst-fm-sound-match

Here are some examples:
https://spiegelib.github.io/spiegelib/examples/fm_sound_match_pages/fm_sound_match_listen.html#fm-sound-match-listen

[RK11111111111]

I think the Genetic algorithm one would be feasible on Rpi hardware. The others will likely be difficult to implement and distribute even on conventional hardware. Tensor flow type neural networks often leverage special hardware.

I saw a commercial VST a while ago that use the Genetic algorithm technique. It had a really cool interface where it would generate approximations, and the user could guide the algorithm for example choosing the closest approximation. Synplant is on my wish list for synths: https://www.youtube.com/watch?v=nIllwZc6BIE

I saw that machine learning algorithm before. I think if I recall, the machine algorithm was able to emulate the same sounds on the DX7, but used fewer OPs than humans used. Like it only needed three operators to do what humans were doing with seven.

[Ma5onic]

@RK11111111111, I checked the TensorFlow documentation on using embedded devices for inference and it is possible to run Deep Learning Models on the Raspberry Pi if the TensorFlow models are converted to "TensorFlow lite" format, as long as the model is using compatible ML operators.
As an optional, additional step, you could compile the TF Lite model for a TPU accelerator (such as the coral.ai TPUs), which can be attached to the RPI for hardware acceleration (for supported operations).

@probonopd If I understand correctly, I think you are looking for something similar to the genopatch feature in "synplant 2" vst by Sonic Charge, where the input is an audio file and the model's output configures the synth parameters to match the timbre of the input audio.
The SpiegeLib seems to have a very useful SynthVST class that can be used to load a VST, Display & Modify the synth parameters, generate random patches, and render the audio for further processing:
https://spiegelib.github.io/spiegelib/reference/synth/synth_vst.html
https://github.com/spiegelib/spiegelib/tree/main/src/spiegelib/synth

Further Reading: Research paper that inspired the creation of SpiegeLib

Interesting DDSP vst projects (similar to the DDX7 method) by google's "magenta" team:
Differentiable Digital Signal Processing - Python Library: https://github.com/magenta/ddsp
Realtime DDSP Neural Synthesizer and Effect VST (Uses TF Lite models): https://github.com/magenta/ddsp-vst

[diyelectromusic]

(sorry for the off-topic prod - but @probonopd did you see that email I sent you? I'm struggling to find a way to get in touch :))

[Ma5onic]

@probonopd, So I took a look at the source code for the experiments that you shared (https://github.com/spiegelib/vst-fm-sound-match) and tried to get it working. I'm still trying to get renderman (spiegelib dependency used to load and control the VST) compiled for my OSs (Windows & Linux). The instructions that are by spiegelib provided are for Mac OSX, but I just noticed that renderman also has.

I've made GUIs for music AIs before and would be willing to help. The only missing component in the spiegelib experiment that you shared, is the ability to save the preset in the VSTs. If that feature isn't available in Renderman or spiegelib natively, we'd need a script to convert the patch parameters used by renderman to midi mappings that configure VST, so that you can save it directly from your VST in the format that it expects.

[Ma5onic]

Spotify seems to have an actively developed library (pedalboard) that has similar functionalities as renderman. I'm looking through the issues and there are some promising leads:
Save and Load Presets Automatically for VST3 Plugins? - spotify/pedalboard#187

[RK11111111111]

Then there is the issue of how to get audio on to the Pi itself. I think it would be interesting to have audio as an Additional 'Operator'.

FM8 does this in a DAW, where you can use audio as a modulator.

[Ma5onic]

@probonopd, What exactly do you mean by DX7 compatible voices? is there code that I can look at to understand the format? I have yet to actually set up a MiniDexed, I just came across this discussion and found it interesting.
I took a look at the source code of the library used by spiegelib to control the vsts (Renderman), but unfortunately the master branch is currently broken and it doesn't currently have the ability to save your preset.
However, I did manage to find a fix for it, which can be found here:
https://github.com/Ma5onic/RenderMan

Instead of creating our own version of renderman, I did find a spiegelib branch that uses DawDreamer which has the ability to save the plugin state to a file then load it back (like a preset) and has the ability to show the UI. This means that after the AI has finished tuning the synth to replicate the input audio, you can save the state, then use the open_editor() function to show the UI, where you could load the state while the plugin UI is open. That would allow you to save the preset in the format used by Dexed.
spiegelib/spiegelib@main...dawdreamer

[Ma5onic]

I've also been thinking of doing something similar with a wavetable synth such as vital with the goal of creating a bare metal Raspberry Pi synth.

Inspiration:

[RK11111111111]

I like Vital. I have been working on a way to run Vital on a raspberry pi as my third rpi project(I just finished making a mixxx DJ console with a custom Rpi touchscreen skin). Vital is compiled for lv2 and it runs on Raspberry Pi OS. I was able to get it up and running using Carla and Jack as a plugin rack. So, all of the pieces are there. Though Carla ran Vital without the 3D gui. I am waiting for my next Rpi in the mail, I ended up using the one for the mixxx console. I like minidexed, but as I mentioned in other threads dexed is geared towards automated envelopes, I want to introduce my kids to real time knob tweaking as an art form.

[Ma5onic]

@RK11111111111 Why do you need carla and jack? I thought that Vital can also run as a standalone app.

[RK11111111111]

I guess technically I was using vitalium not vital. Vitalium is a fork that supports linux arm compilation. It looks like there is a standalone version: https://github.com/jackreeds/vitalium-standalone

[Ma5onic]

Sick! I never heard of that project but that's awesome! thanks for sharing :)

[RK11111111111]

It doesn't appear to be compiling on my pi 4. I think it has to be cross compiled.

[Ma5onic]

This repo looks very promising:
https://github.com/dafaronbi/Multi-Task-Automatic-Synthesizer-Programming
Examples:
https://dafaronbi.github.io/Multi-Task-Automatic-Synthesizer-Programming/
Video - Exploring Approaches to Multi-Task Automatic Synthesizer Programming:
https://youtu.be/LvCntIlJpxM

[probonopd]

https://github.com/Sound2Synth/Sound2Synth also looks promising.
It needs to be trained. Would be nice if a pre-trained model would be published.