6 ‐ Figures, Images, Lists

The Musical Gestures Toolbox includes several tools to extract data from audio-visual content, and many of these tools output figures (or images) to visualize this time-varying data. In this chapter we take a closer look on how we can customize and combine figures and images from the toolbox.

Titles

By default the figures rendered by the toolbox automatically get the title of the source file we analyzed. We can also change this by providing a title as an argument to the function or method we are using. Here are some examples:

source_video = musicalgestures.MgVideo('/path/to/source/video.avi') # source video as an MgVideo
motionplots = source_video.motionplots(title='My Video - Motion Plots') # motion plots
spectrogram = source_video.audio.spectrogram(title='My Video - Spectrogram') # spectrogram
tempogram = source_video.audio.tempogram(title='My Video - Tempogram') # tempogram
descriptors = source_video.audio.descriptors(title='My Video - Spectral Descriptors') # descriptors

Helper classes: MgFigure, MgImage, and MgList

When we work with video files in musicalgestures we almost always use MgVideo, preprocess the videos via the objects' arguments and apply further processes via class methods. We use similar helper classes for working with Matplotlib figures, and image files. MgFigure is our way to make matplotlib figures reusable and modular, and MgImage serves a similar purpose when it comes to images. Additionally, we also have our extended list implementation with MgList that replicates more-or-less the same functionality as normal Python lists, but with some methods that make it easy to stack MgFigures, MgImages, or MgLists of these.

MgFigure

MgFigure offers an organized, comfortable way to represent the type of the figure and its data, so that we can reuse it in other figures. It implements the show method which is used across the musicalgestures package to show the content of an object. In the case of MgFigure this will show the internal matplotlib.pyplot.figure object.

source_video = musicalgestures.MgVideo('/path/to/source/video.avi') # source video as an MgVideo
spectrogram = source_video.audio.spectrogram() # spectrogram
spectrogram.show() # show internal figure
spectrogram.figure # this also works

What is more important is that each MgFigure has a figure_type attribute. This is what you see when you print (or in a notebook, evaluate) them:

source_video = musicalgestures.MgVideo('/path/to/source/video.avi')
spectrogram = source_video.audio.spectrogram()
print(spectrogram) # MgFigure(figure_type='audio.spectrogram')

Each MgFigure object has a data attribute, where we store all the related data to be able to recreate the figure elsewhere. You never really have to interact with this attribute directly, unless you want to look under the hood:

# see what kind of entries we have in our spectrogram figure
print(spectrogram.data.keys())  # dict_keys(['hop_size', 'sr', 'of', 'S', 'length'])
# see the length (in seconds) of the source file
print(spectrogram.data['length']) # 97.62

You can also get the rendered image file corresponding to the MgFigure object.

print(spectrogram.image) # MgImage('path/to/the/image.png')

Another attribute of MgFigure is called layers. We will get back to this in a bit, for now let's just say that when an MgFigure object is in fact a composition of other MgFigure, MgImage or MgList objects, we have access to all those in the layers of the "top-level" MgFigure.

MgImage

MgImage is a simple class that takes a filename (path) on initialization, and implements the show method that is used across musicalgestures.

img = musicalgestures.MgImage('path/to/the/image.png') # create an MgImage object
img.show() # shows the image in a separate window
img.show(mode="notebook") # embeds the image in a jupyter notebook
print(img) # MgImage('path/to/the/image.png')
print(img.filename) # path/to/the/image.png
print(img.of) # path/to/the/image
print(img.fex) # .png

MgList

Another versatile tool in our hands is MgList. It works more-or-less as an ordinary list, and it is specifically designed for working with objects of the musicalgestures package. Here is an example how to use it:

source_video = musicalgestures.MgVideo('/path/to/source/video.avi')
spectrogram = source_video.audio.spectrogram() # create a spectrogram (returns an MgFigure)
tempogram = source_video.audio.tempogram() # create a tempogram (returns an MgFigure)
my_list = musicalgestures.MgList(spectrogram, tempogram) # create an MgList of the two MgFigures
print(my_list) # MgList('[MgFigure(figure_type='audio.spectrogram'), MgFigure(figure_type='audio.tempogram')]')

MgList also implements many of the list feaures you already know:

# how many objects are there?
print(f'There are {len(my_list)} objects in this MgList!') # There are 2 objects in this MgList!

# which one is the 2nd?
print(f'The second object is a(n) {my_list[1]}.') # The second object is a(n) MgFigure(figure_type='audio.tempogram').

# change the 2nd element to a descriptors figure instead:
descriptors = source_video.audio.descriptors() # create an MgFigure with the audio descriptors
my_list[1] = descriptors
print(f'The second object is now {my_list[1]}.') # The second object is now MgFigure(figure_type='audio.descriptors').

# add the tempogram figure to the list:
my_list += tempogram
print(f'Now there are {len(liszt_list)} objects in this MgList. These are:')
for element in my_list:
    print(element)
# Now there are 3 objects in this MgList. These are:
# MgFigure(figure_type='audio.spectrogram')
# MgFigure(figure_type='audio.descriptors')
# MgFigure(figure_type='audio.tempogram')

# fun fact: videograms() returns an MgList with the horizontal and vertical videograms (as MgImages)
videograms = source_video.videograms()
print(videograms) # MgList('[MgImage('path/to/horizontal/videogram.png'), MgImage('path/to/vertical/videogram.png')]')

# MgList.show() will call show() on all its objects in a succession
videograms.show() # will show both videograms

# add two MgLists
everything = videograms + my_list
print('everything:', everything) # everything: MgList('[MgImage('path/to/horizontal/videogram.png'), MgImage('path/to/vertical/videogram.png'), MgFigure(figure_type='audio.spectrogram'), MgFigure(figure_type='audio.descriptors'), MgFigure(figure_type='audio.tempogram')]')

Combining figures and images

One of the most useful methods of MgList is as_figure. It allows you to conveniently compose a stack of plots, time-aligned, and with a vertical order you specify.

The workflow:

1. Create the MgFigures and MgImages (or MgLists of these) you want to stack together,
2. Collect all of them into an MgList,
3. Call as_figure. The first element in the Mglist will correspond to the top figure in the stack, and the last element to the bottom figure.

Here is an example (following the code above):

fig_everything = everything.as_figure(title='Liszt - Mephisto Waltz No. 1')

A time-aligned figure-stack