Similarity (Essentia) DSTM mixer - call for help

**Roland0** · 2021-01-13, 01:11

Code:

* seed: /data/music/Philip Glass - Koyaanisqatsi/03 - Cloudscape.flac
+----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+
| file                                                                                         |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |   bpm |
|----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------|
| /data/music/Philip Glass - Koyaanisqatsi/03 - Cloudscape.flac                                |       0.040 |        0.003 |        0.698 |      0.286 |   0.301 |   0.026 |     0.995 | 0.331 |  0.984 |   0.039 |   0.003 |   134 |
| /data/music/Max Richter - The Blue Notebooks/02 - On The Nature Of Daylight.flac             |       0.338 |        0.085 |        0.629 |      0.555 |   0.034 |   0.048 |     0.988 | 0.428 |  0.971 |   0.000 |   0.000 |   123 |
| /data/music/The Dead Texan - The Dead Texan/08 - Girth Rides A (Horse) +.mp3                 |       0.007 |        0.014 |        0.173 |      0.960 |   0.436 |   0.016 |     0.983 | 0.844 |  0.902 |   0.992 |   0.000 |   135 |
| /data/music/The Dead Texan - The Dead Texan/02 - Glen's Goo.mp3                              |       0.003 |        0.061 |        0.351 |      0.601 |   0.105 |   0.016 |     0.967 | 0.745 |  0.914 |   0.950 |   0.000 |   129 |
| /data/music/SPK - Zamia Lehmanni (Songs Of Byzantine Flowers)/05 - In Flagrante Delicto.flac |       0.006 |        0.161 |        0.396 |      0.990 |   0.019 |   0.003 |     1.000 | 0.500 |  0.991 |   0.681 |   0.003 |   114 |
| /data/music/Rachel's - The Sea And The Bells/04 - Lloyd's Register.flac                      |       0.027 |        0.015 |        0.740 |      0.832 |   0.271 |   0.015 |     0.997 | 0.377 |  0.915 |   0.992 |   0.000 |   154 |
| /data/music/Philip Glass - Koyaanisqatsi OST/03 - Clouds.flac                                |       0.355 |        0.000 |        0.728 |      0.430 |   0.735 |   0.240 |     0.874 | 0.500 |  0.307 |   0.506 |   0.000 |   132 |
| /data/music/Amber Asylum - Frozen in Amber/04 - Heckle and Jeckle.flac                       |       0.279 |        0.080 |        0.789 |      0.310 |   0.000 |   0.080 |     0.909 | 0.436 |  0.977 |   0.005 |   0.000 |   147 |
| /data/music/Rachels - Handwriting/05 - Seratonin.mp3                                         |       0.009 |        0.094 |        0.267 |      0.991 |   0.102 |   0.002 |     0.972 | 0.756 |  0.983 |   0.955 |   0.000 |   148 |
+----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+
*** stats
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+
| func   |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |     bpm |
|--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------|
| min    |       0.003 |        0.000 |        0.173 |      0.310 |   0.000 |   0.002 |     0.874 | 0.377 |  0.307 |   0.000 |   0.000 | 114.000 |
| max    |       0.355 |        0.161 |        0.789 |      0.991 |   0.735 |   0.240 |     1.000 | 0.844 |  0.991 |   0.992 |   0.003 | 154.000 |
| mean   |       0.128 |        0.064 |        0.509 |      0.709 |   0.213 |   0.052 |     0.961 | 0.573 |  0.870 |   0.635 |   0.000 | 135.250 |
| median |       0.018 |        0.070 |        0.513 |      0.717 |   0.103 |   0.016 |     0.978 | 0.500 |  0.943 |   0.816 |   0.000 | 133.500 |
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+
*** delta (track - seed track)
+----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+
| file                                                                                         |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |   bpm |
|----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------|
| /data/music/Max Richter - The Blue Notebooks/02 - On The Nature Of Daylight.flac             |       0.297 |        0.082 |       -0.068 |      0.269 |  -0.267 |   0.023 |    -0.007 | 0.097 | -0.013 |  -0.039 |  -0.003 |   -11 |
| /data/music/The Dead Texan - The Dead Texan/08 - Girth Rides A (Horse) +.mp3                 |      -0.033 |        0.011 |       -0.525 |      0.674 |   0.135 |  -0.010 |    -0.012 | 0.514 | -0.082 |   0.953 |  -0.003 |     1 |
| /data/music/The Dead Texan - The Dead Texan/02 - Glen's Goo.mp3                              |      -0.037 |        0.058 |       -0.347 |      0.316 |  -0.196 |  -0.010 |    -0.028 | 0.414 | -0.069 |   0.911 |  -0.003 |    -5 |
| /data/music/SPK - Zamia Lehmanni (Songs Of Byzantine Flowers)/05 - In Flagrante Delicto.flac |      -0.034 |        0.157 |       -0.302 |      0.704 |  -0.282 |  -0.022 |     0.005 | 0.169 |  0.007 |   0.643 |  -0.000 |   -20 |
| /data/music/Rachel's - The Sea And The Bells/04 - Lloyd's Register.flac                      |      -0.013 |        0.012 |        0.042 |      0.546 |  -0.030 |  -0.010 |     0.002 | 0.046 | -0.069 |   0.953 |  -0.003 |    20 |
| /data/music/Philip Glass - Koyaanisqatsi OST/03 - Clouds.flac                                |       0.315 |       -0.003 |        0.030 |      0.144 |   0.434 |   0.214 |    -0.121 | 0.169 | -0.677 |   0.467 |  -0.003 |    -2 |
| /data/music/Amber Asylum - Frozen in Amber/04 - Heckle and Jeckle.flac                       |       0.239 |        0.076 |        0.091 |      0.024 |  -0.301 |   0.054 |    -0.087 | 0.105 | -0.007 |  -0.033 |  -0.003 |    13 |
| /data/music/Rachels - Handwriting/05 - Seratonin.mp3                                         |      -0.032 |        0.091 |       -0.431 |      0.705 |  -0.199 |  -0.023 |    -0.023 | 0.426 | -0.001 |   0.916 |  -0.003 |    14 |
+----------------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+
*** stats (delta)
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+
| func   |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |     bpm |
|--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------|
| min    |      -0.037 |       -0.003 |       -0.525 |      0.024 |  -0.301 |  -0.023 |    -0.121 | 0.046 | -0.677 |  -0.039 |  -0.003 | -20.000 |
| max    |       0.315 |        0.157 |        0.091 |      0.705 |   0.434 |   0.214 |     0.005 | 0.514 |  0.007 |   0.953 |  -0.000 |  20.000 |
| mean   |       0.088 |        0.060 |       -0.189 |      0.423 |  -0.088 |   0.027 |    -0.034 | 0.242 | -0.114 |   0.596 |  -0.003 |   1.250 |
| median |      -0.023 |        0.067 |       -0.185 |      0.431 |  -0.198 |  -0.010 |    -0.017 | 0.169 | -0.041 |   0.777 |  -0.003 |  -0.500 |
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+

**cpd73** · 2021-01-13, 01:20

Originally Posted by Roland0

Hmmm:

Code:

* seed: /data/music/Pixies - Brixton Academy 05.06.2004/1 - 03 - Monkey Gone To Heaven.mp3
+------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+
| file                                                                               |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |   bpm |
|------------------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------|
| /data/music/Pixies - Brixton Academy 05.06.2004/1 - 03 - Monkey Gone To Heaven.mp3 |       0.146 |        0.574 |        0.661 |      0.037 |   0.348 |   0.281 |     0.634 | 0.487 |  0.750 |   0.151 |   0.836 |   119 |
| /data/music/Pixies - Brixton Academy 02.06.2004/1 - 08 - Monkey Gone To Heaven.mp3 |       0.044 |        0.000 |        0.595 |      0.024 |   0.570 |   0.000 |     0.809 | 0.480 |  0.915 |   0.544 |   0.809 |   121 |
| /data/music/Pixies - Coachella 2004/11 - Monkey Gone To Heaven.mp3                 |       0.569 |        0.326 |        0.536 |      0.186 |   0.247 |   0.110 |     0.590 | 0.692 |  0.775 |   0.809 |   0.999 |   124 |

I assume these are all basically the same track? Odd that essentia has very different 'aggressive' values. How different are the recordings? Any reason why the 1st two would be so different in their agressive attribute?

Also, when looking at differences I'm looking at the absolute difference - so -2 and 2 are the same. (This migh tbe a bad thing to do, (as you can tell) I'm not really sure what I'm doing here...)

**Roland0** · 2021-01-18, 01:08

Originally Posted by cpd73

I assume these are all basically the same track? Odd that essentia has very different 'aggressive' values. How different are the recordings? Any reason why the 1st two would be so different in their agressive attribute?

they are different live versions of the same song, and very similar (none is acoustic only, has bad audio quality or a noisy audience etc.), so there's no obvious explanation for the variance.
looking at all versions of this song it's even worse:

Code:

0.5
0.605293393135
0.137314662337
0.688453316689
0.709211885929
0.919100821018
0.830868721008
0.326436012983
0.614248454571
0.573822319508
1.96212039671e-11
0.38925370574

the tensorflow results seem to be more consistent:

Code:

json.metadata.tags.file_name = "11 - Monkey Gone To Heaven.mp3";
json.highlevel["esstf.mood_aggressive.musicnn-mtt"].all.aggressive = 0.12698836624622345;
json.metadata.tags.file_name = "1 - 03 - Monkey Gone To Heaven.mp3";
json.highlevel["esstf.mood_aggressive.musicnn-mtt"].all.aggressive = 0.25417667627334595;
json.metadata.tags.file_name = "1 - 08 - Monkey Gone To Heaven.mp3";
json.highlevel["esstf.mood_aggressive.musicnn-mtt"].all.aggressive = 0.13030590116977692;

Also, when looking at differences I'm looking at the absolute difference - so -2 and 2 are the same. (This migh tbe a bad thing to do, (as you can tell) I'm not really sure what I'm doing here...)

Meanwhile, I've added some similarity metrics to the essentia / musly comparison (approach is rather naive: track is a 12-dim vector, calculate euclidean distance, manhattan distance and cosine similarity):

Code:

* seed: /data/music/Pixies - Brixton Academy 05.06.2004/1 - 03 - Monkey Gone To Heaven.mp3
+------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+----------+----------+----------+
| file                                                                   |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |   bpm |   eucdst |   mandst |   cossim |
|------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+----------+----------+----------|
| Pixies - Brixton Academy 05.06.2004/1 - 03 - Monkey Gone To Heaven.mp3 |       0.146 |        0.574 |        0.661 |      0.037 |   0.348 |   0.281 |     0.634 | 0.487 |  0.750 |   0.151 |   0.836 |   119 |          |          |          |
| Pixies - Brixton Academy 02.06.2004/1 - 08 - Monkey Gone To Heaven.mp3 |       0.044 |        0.000 |        0.595 |      0.024 |   0.570 |   0.000 |     0.809 | 0.480 |  0.915 |   0.544 |   0.809 |   121 |    0.828 |    2.038 |    0.899 |
| Pixies - Coachella 2004/11 - Monkey Gone To Heaven.mp3                 |       0.569 |        0.326 |        0.536 |      0.186 |   0.247 |   0.110 |     0.590 | 0.692 |  0.775 |   0.809 |   0.999 |   124 |    0.908 |    2.352 |    0.896 |
| Pixies - Brixton Academy 04.06.2004/1 - 13 - No 13 Baby.mp3            |       0.445 |        0.028 |        0.721 |      0.059 |   0.450 |   0.136 |     0.818 | 0.245 |  0.660 |   0.892 |   0.578 |   123 |    1.068 |    2.718 |    0.828 |
| Pixies - Brixton Academy 02.06.2004/1 - 11 - No. 13 Baby.mp3           |       0.000 |        0.000 |        0.833 |      0.017 |   0.322 |   0.000 |     0.809 | 0.399 |  0.728 |   0.944 |   0.713 |   127 |    1.070 |    2.477 |    0.839 |
| Pixies - Eugene 28-04-2004/13 - Monkey Gone To Heaven.mp3              |       0.102 |        0.605 |        0.086 |      0.005 |   0.227 |   0.427 |     0.316 | 0.165 |  0.225 |   0.163 |   0.954 |   119 |    0.930 |    2.246 |    0.852 |
| Pixies - Brixton Academy 03.06.2004/2 - 10 - No 13 Baby - Caribou.mp3  |       0.316 |        0.009 |        0.836 |      0.082 |   0.065 |   0.046 |     0.887 | 0.405 |  0.959 |   0.319 |   0.068 |   126 |    1.118 |    3.006 |    0.797 |
| Pixies - Brixton Academy 05.06.2004/2 - 02 - Where Is My Mind.mp3      |       0.353 |        0.081 |        0.276 |      0.312 |   0.900 |   0.209 |     0.337 | 0.410 |  0.111 |   0.969 |   0.815 |   162 |    1.449 |    4.161 |    0.693 |
| Pixies - Brixton Academy 05.06.2004/1 - 14 - Gouge Away.mp3            |       0.616 |        0.429 |        0.244 |      0.060 |   0.658 |   0.473 |     0.035 | 0.643 |  0.500 |   0.881 |   0.508 |   124 |    1.279 |    3.658 |    0.742 |
| Pixies - Regina, SK, Canada/1 - 11 - Where Is My Mind.mp3              |       0.457 |        0.217 |        0.159 |      0.025 |   0.601 |   0.184 |     0.196 | 0.305 |  0.023 |   0.995 |   0.514 |   155 |    1.481 |    4.318 |    0.618 |
| Pixies - Brixton Academy 05.06.2004/1 - 11 - I Bleed.mp3               |       0.430 |        0.824 |        0.422 |      0.045 |   0.794 |   0.370 |     0.390 | 0.167 |  0.943 |   0.091 |   0.987 |    92 |    0.822 |    2.492 |    0.910 |
+------------------------------------------------------------------------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+-------+----------+----------+----------+
*** stats
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+----------+----------+----------+
| func   |   danceable |   aggressive |   electronic |   acoustic |   happy |   party |   relaxed |   sad |   dark |   tonal |   voice |     bpm |   eucdst |   mandst |   cossim |
|--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+----------+----------+----------|
| min    |       0.000 |        0.000 |        0.086 |      0.005 |   0.065 |   0.000 |     0.035 | 0.165 |  0.023 |   0.091 |   0.068 |  92.000 |    0.822 |    2.038 |    0.618 |
| max    |       0.616 |        0.824 |        0.836 |      0.312 |   0.900 |   0.473 |     0.887 | 0.692 |  0.959 |   0.995 |   0.999 | 162.000 |    1.481 |    4.318 |    0.910 |
| mean   |       0.333 |        0.252 |        0.471 |      0.082 |   0.484 |   0.195 |     0.519 | 0.391 |  0.584 |   0.661 |   0.695 | 127.300 |    1.095 |    2.947 |    0.807 |
| median |       0.392 |        0.149 |        0.479 |      0.052 |   0.510 |   0.160 |     0.490 | 0.402 |  0.694 |   0.845 |   0.761 | 124.000 |    1.069 |    2.605 |    0.833 |
+--------+-------------+--------------+--------------+------------+---------+---------+-----------+-------+--------+---------+---------+---------+----------+----------+----------+

Not really promising, so I'll abandon this approach (another thing to consider is that musly bases similarity on timbre only (melbands), while essentia uses timbral as well as temporal features (at least the tensorflow version does, haven't checked SVM), so this may not have been such a great idea in the first place)
What I'll try next is to use k-nearest neighbors on the essentia data and see if this finds similar tracks.

**Roland0** · 2021-01-27, 00:27

Originally Posted by cpd73

Great! Can you detail this in the Essentia thread in the dev section?

I've modified LMS Musly to use a nearest neighbour search on the essentia classifications. Tracks are modeled as 13d vectors (essentia attributes + normalized bpm + genres mapped on 0-1)
Seems to basically work, needs more testing, Currently, it seems that any obvious mismatches are due to dubious essentia results rather than the algorithm.

Some other issues:
Results are 100% deterministic using a fixed seed track, so I'm thinking about adding some randomization option ( maybe varying the seed track, or using an approximate nearest neighbour search) for dynamic playlists.
Also, I'd like to avoid having several different versions of the same track in a sequence

I did make a change to use "Euclidean distance" - but I was more experimenting, as I've never used / thought-about this stuff before.

How well does this work (the version on github seems not to include this)?

**cpd73** · 2021-01-27, 01:21

Originally Posted by Roland0

I've modified LMS Musly to use a nearest neighbour search on the essentia classifications.

'Musly'? Do you not mean 'LMS Essentia' ? Where can I find the source, with your changes? I cant see it on the Musly or Essentia pages.

Originally Posted by Roland0

How well does this work (the version on github seems not to include this)?

...I think it works better, but as always that's very subjective. Its on github (latest master), with this commit: https://github.com/CDrummond/essenti...378817a61e8b74 ...assuming my understanding is correct. Each attribute difference is squared, these are all added together, and then the squareroot taken.

**Roland0** · 2021-01-28, 01:47

Originally Posted by cpd73

'Musly'? Do you not mean 'LMS Essentia' ?

No, I'm writing a stand-alone app for static/dynamic playlists/API (i.e. what LMS Musly provides), but with a different similarity method (nearest neighbour search on the essentia classifications instead of musly).

Where can I find the source, with your changes? I cant see it on the Musly or Essentia pages.

It's still WIP, I'll probably release it in a few days.

**cpd73** · 2021-01-28, 02:18

Originally Posted by Roland0

nearest neighbour search on the essentia classifications instead of musly

How is that different to the euclidean distance? From my, very limited, knowledge I'm assuing you get the euclidean distance for all items, sort, and use the nearest few. Which is what my latest code does. How do you 'normalise' bpm? I've taken it as 'abs(seed_bpm-candidate_bpm)/seed_bpm' And for genre, no genre in seed or candidate the 0.7, same genre 0.3, differnet genre 0.9, similar genre 0.5 (for 'similar' genre I have a list of genre sets (so "rock, metal", etc).

**Roland0** · 2021-01-29, 02:25

Originally Posted by cpd73

How is that different to the euclidean distance?

euclidean distance can be used as a distance metric for nearest neighbor search

From my, very limited, knowledge I'm assuing you get the euclidean distance for all items, sort, and use the nearest few.

I'm using a k-d tree

Which is what my latest code does.

From your description, seems to be a linear search

How do you 'normalise' bpm? I've taken it as 'abs(seed_bpm-candidate_bpm)/seed_bpm'

assuming normalizing 0-1: x normalized = (x – x minimum) / (x maximum – x minimum)

And for genre, no genre in seed or candidate the 0.7, same genre 0.3, differnet genre 0.9, similar genre 0.5 (for 'similar' genre I have a list of genre sets (so "rock, metal", etc).

that's more or less what I do as well

**cpd73** · 2021-01-29, 14:02

Originally Posted by Roland0

I'm using a k-d tree

From your description, seems to be a linear search

I assume a k-d tree would be faster? Would that require storing contents of all in memory? I re-wrote my search to perform mostly in SQLite, take the top 2500 rows, add genre, then re-sort. This takes a total of ~360ms on my pi4 for around 20k songs. So, not bad speed wise, and fast enough - I think.

**Roland0** · 2021-01-30, 14:38

Originally Posted by cpd73

I assume a k-d tree would be faster?

Yes. My query function returns the n nearest tracks (query includes bpm and genre), sorted by similarity, no further processing, so ~ your query time.
Tested on an odroid hc2, single (big) core, probably ~ same performance as RPi4
Database has 3500 tracks

Code:

** Querying for 100 tracks:
2021-01-30 21:52:00 DEBUG    Query time:73
2021-01-30 21:52:00 DEBUG    Total time:233
(50 tracks returned)

DEBUG:lmsessim.lib.tracks_source:total time: 6
(100 tracks returned)

** Querying for 20 tracks:
2021-01-30 21:59:21 DEBUG    Query time:71
2021-01-30 21:59:21 DEBUG    Total time:230
(20 tracks returned)

DEBUG:lmsessim.lib.tracks_source:total time: 4
(20 tracks returned)

Would that require storing contents of all in memory?

Yes, but I keep all track data in memory anyway. Not really an issue:

Code:

DEBUG:__main__:data memory size: features: 0.17 MB / labels: 0.03 MB

I re-wrote my search to perform mostly in SQLite, take the top 2500 rows, add genre, then re-sort. This takes a total of ~360ms on my pi4 for around 20k songs. So, not bad speed wise, and fast enough - I think.

I'd agree, performance seems not to be a real concern.
However, there are other advantages as well:
- I don't have to write the code myself
- I can fairly simply use different distance metrics and evaluate if this helps quality (tbh, probably only of academic interest, I don't expect significant (any?) improvements)

Thread: Similarity (Essentia) DSTM mixer - call for help

Thread Tools

Search Thread

Display

Posting Permissions