Serialization efficiency

[cifkao]

Below is a little benchmark of the efficiency of YAML, JSON and other representations on a file from NES-MDB.

Wow!

• MusPy YAML and JSON files are big
• BSON is slower than JSON and almost as big
• Pickling is much faster than anything else and the result is a bit smaller than JSON (but still big compared to the MIDI file)
• Gzipped JSON can be even smaller than MIDI and not much slower to write than plain JSON

Conclusion: Support for gzipped JSON should be added 🙂

%timeit music.write_midi(out_path.with_suffix('.mid'))

9.55 ms ± 8.33 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit music.save(out_path.with_suffix('.json'))

5.95 ms ± 8.15 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
with open(out_path.with_suffix('.bson'), 'wb') as f:
    f.write(bson.dumps(music.to_ordered_dict()))

7.84 ms ± 8.42 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
with gzip.open(out_path.with_suffix('.json.gz'), 'wt') as f:
    json.dump(music.to_ordered_dict(), f)

8.04 ms ± 6.66 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
with lzma.open(out_path.with_suffix('.json.xz'), 'wt') as f:
    json.dump(music.to_ordered_dict(), f)

22 ms ± 94 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit music.save(out_path.with_suffix('.yaml'))

96.9 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
with open(out_path.with_suffix('.muspy.pkl'), 'wb') as f:
    pickle.dump(music, f)

500 µs ± 423 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
with open(out_path.with_suffix('.note-seq.bin'), 'wb') as f:
    f.write(note_seq.midi_to_sequence_proto(music.to_pretty_midi()).SerializeToString())

2.63 ms ± 1.44 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

!ls -lSrh {testdir}

total 88K
-rw-r--r-- 1 ocifka tsi 1.9K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.json.xz
-rw-r--r-- 1 ocifka tsi 2.2K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.json.gz
-rw-r--r-- 1 ocifka tsi 2.3K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.mid
-rw-r--r-- 1 ocifka tsi 7.6K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.note-seq.bin
-rw-r--r-- 1 ocifka tsi  12K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.muspy.pkl
-rw-r--r-- 1 ocifka tsi  16K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.bson
-rw-r--r-- 1 ocifka tsi  18K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.json
-rw-r--r-- 1 ocifka tsi  18K Dec 27 14:23 002_1943_TheBattleofMidway_00_01Title.yaml

[cifkao]

Same benchmark for a small file (tests/data/test.json). Here MIDI and note-seq (protocol buffer) clearly win over compressed formats in terms of size.

%timeit music.write_midi(out_path.with_suffix('.mid'))

626 µs ± 1.04 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit music.save(out_path.with_suffix('.json'))

585 µs ± 4.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
with open(out_path.with_suffix('.bson'), 'wb') as f:
    f.write(bson.dumps(music.to_ordered_dict()))

747 µs ± 1.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
with gzip.open(out_path.with_suffix('.json.gz'), 'wt') as f:
    json.dump(music.to_ordered_dict(), f)

798 µs ± 883 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
with lzma.open(out_path.with_suffix('.json.xz'), 'wt') as f:
    json.dump(music.to_ordered_dict(), f)

4.13 ms ± 5.63 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit music.save(out_path.with_suffix('.yaml'))

6.76 ms ± 12.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
with open(out_path.with_suffix('.muspy.pkl'), 'wb') as f:
    pickle.dump(music, f)

126 µs ± 212 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%%timeit
with open(out_path.with_suffix('.note-seq.bin'), 'wb') as f:
    f.write(note_seq.midi_to_sequence_proto(music.to_pretty_midi()).SerializeToString())

281 µs ± 371 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

!ls -lSrh {testdir}

total 32K
-rw-r--r-- 1 ocifka tsi  192 Dec 27 14:25 example.mid
-rw-r--r-- 1 ocifka tsi  259 Dec 27 14:25 example.note-seq.bin
-rw-r--r-- 1 ocifka tsi  415 Dec 27 14:25 example.json.gz
-rw-r--r-- 1 ocifka tsi  460 Dec 27 14:25 example.json.xz
-rw-r--r-- 1 ocifka tsi 1.2K Dec 27 14:25 example.yaml
-rw-r--r-- 1 ocifka tsi 1.2K Dec 27 14:25 example.json
-rw-r--r-- 1 ocifka tsi 1.3K Dec 27 14:25 example.bson
-rw-r--r-- 1 ocifka tsi 1.5K Dec 27 14:25 example.muspy.pkl

[salu133445]

This is interesting. Thanks for testing this. Our main reason to choose JSON and YAML is for readability. However, it would be also great to support other formats for speed and memory reason.

Some ideas:

• We could add a compress option (boolean) to save_json and save_yaml for gzipped JSON and YAML.
• We could support converting into note-seq (protobuf) in a way like you did in the experiment.

[cifkao]

A great first step would be to allow passing open files instead of paths to save and load methods/functions.

We could support converting into note-seq (protobuf) in a way like you did in the experiment.

Yes, though this is not lossless.

BTW, I also tested read speed. The results are similar, except that:

• LZMA (.xz) is slow for writing, but fast for reading (comparable to gzip, and the compression ratio is better)
• JSON is also faster for reading than writing

[cifkao]

One more idea: an ArchiveDataset, like FolderDataset but reading from a (possibly compressed) tarball. This should provide better compression if the dataset consists of a large number of small files.