AV-Sync

Builds a beat-synced music video from an input audio track and a folder of source video clips.

Overview

AV-Sync analyzes source audio files with Librosa, separates harmonic and percussive content, detects onsets and divides the track into musical segments using different segmentation modes.

For each audio segment, AV-Sync computes detailed rhythmic, loudness and novelty features, then classifies the segment as slow, medium or fast. It also analyzes each candidate video with OpenCV by sampling motion over time, building timelines and grouping clips into motion buckets so the visual pacing can follow the energy of the music.

Finally, AV-Sync selects diverse video regions to best match each audio segment’s motion profile and timing. It then builds an FFmpeg filter graph which trims clips, applies zooms, pans, pulse effects, motion trails and audio mapping before rendering the final synchronized video.

The program also writes a JSON manifest containing the audio analysis, clip choices, timing offsets, feature measurements and placement metadata.

Basic Setup Instructions

Below are the required software programs and instructions for installing and using this application on a Linux machine.

Programs Needed

• Git

• Python

Steps For Use

1. Install the above programs

2. Open a terminal

3. Clone this repository: git clone git@github.com:devbret/av-sync.git

4. Navigate to the repo's directory: cd av-sync

5. Create a virtual environment: python3 -m venv venv

6. Activate your virtual environment: source venv/bin/activate

7. Install the needed dependencies: pip install -r requirements.txt

8. Place your source files into the input and videos directories

9. Run the program: python3 app.py

10. Locate your results in the output directory

11. Exit the virtual environment: deactivate

Other Considerations

Below you will find information not covered in the installation and use sections above. Including the abilities this repo is intended to demonstrate. As well as an overview of the license this code is made available with. And a way to contact the maintainer with questions, suggestions and collaboration opportunities.

Abilities Demonstrated

This project repo is intended to demonstrate an ability to do the following:

• Turn an audio track and a folder of video clips into a beat-synced music video

• Analyze tempo, onsets, energy and tonal features of the source audio file

• Score videos by motion and select clip regions to match the pace and intensity of each audio segment

• Save a JSON manifest of the audio analysis, clip choices, timing and placement data

License Information

This repository is distributed under the MIT License. You are free to use, copy, modify, merge, publish, distribute, sublicense and sell copies of this software, including as part of proprietary or commercial work. The single condition is the copyright and permission notices contained in the LICENSE file must be included with any copy or substantial portion of the software that you redistribute. The software is provided "as is", without warranty of any kind, and the copyright holder is not liable for any claim or damages arising from its use.

If you have any questions or would like to collaborate, please reach out either on GitHub or via my website.