Adam Berenzweig

First test: failed

I implemented a rank-agreement score to compare the ranking-by-audio-similarity to the ranking-by-AOTM-cooccurrence. Say we have N item, and two rankings of those items, where a ranking is just a permutation of the items. One of the rankings, say A, is the reference ranking, and the other, say B, is the ranking to be evaluated. For example, for each artist X, A is a list of the artists in decreasing order of conditional co-occurrence probability given artist X. B is the artists ordered by similarity to X under whatever metric we're testing. Note that B doesn't have to a complete ordering, we could take the top 10 hits or something.

The rank-agreement score R is:
R = w'*r
where w is a vector of weights that sum to one and are exponentially decreasing, and r is the "rank permutation" vector: r(i) is the index of element B(i) in A. For example if the sim-based distance puts Eden's Crush as 2nd-closest to reference Aaliyah, but ranked by AOTM co-occurrence, Eden's crush is 100th closest to Aaliyah, then r(2)=100.

The weights are calculated according to the exponential
w = exp(-(log(2)/halflife)*(0:(length(A)-1)));
I used halflife=20 for now.

The optimal value of this metric is when A and B agree completely, i.e. are equal, so then r=1:N. For 414 artists and halflife=20, then optimal_R=29.4. So for the evaluation, I find the rank-agreement score R conditioned on each of the 414 artists, and take opt_R/R for each, and average. So perfection is 1.

To do a statistical significance test (I hope I passed my midterm), we need to know what this score will do under a null hypothesis that the ranking to be evaluated (B) is a random permutation. So I created 10,000 random permutations r of 1:414 and computed w'*r for each. The histogram of that, with the mean in red, is shown here. It looks roughly normal (but a little right-tailed), so I use a basic one-sided significance test with the Normal distribution (probly should use t-test since both mean and variance are unknown, but close enough for now.) The 95% significance point for this test is .1590, but the mean score under the ALA-based SIM metric that I'm using now was .143. Way below 95% significance. In fact, the p-value is about .54, which is completely insignificant.

I also tried normalizing the conditional probability by the prior, to adjust for popularity effects. The test also failed, with p-value .52, even worse than the non-normalized case.

Next:

• I'm not sure that I should be using the entire rank ordering, perhaps only the top 10 or 20 are important? But the halflife should take care of that, downweighting the later ranks. I'll play around with the halflife.
• There are a number of artists that don't appear in the AOTM lists, so the conditional densities for those are empty, and I need to deal with that.
• Finally, I can retrain the GMMs with more iterations to see if that helps, and also perhaps try it at the song leve.

Posted by madadam at March 24, 2003 04:48 PM