Bug 1: I wasnt treating distance matrices differently from similarity matrices.
Bug 2: I didnt understand what I was doing with some index sorting business (cdix) in the ranking eval, so I was getting basically random results.
Now fixed, the results make sense. I threw out the ranking agreement eval and went with the Information Retrieval style eval: treat the top N most probable co-occuring artists in AOTM playlists as ground truth similar hits, and treat the ranked SIM row as "retrieval results". The rank of each hit gets an exponentially decreasing score with halflife 20, then take the mean. So for N=10, optimal is .8598. This gives me a score for each row, and then I take the mean for an overall score.
Using N=10, random permutations score like this histogram. It's not Gaussian (because it bumps up against zero on the left), but I treat it like a Gaussian and do the same confidence test that I tried before. (If I make the cutoff larger, e.g. 20, it starts to look Gaussian, and as cutoff gets bigger the variance shrinks.)
So here are the results using the AOTM IR-type evaluation with N=10 and halflife=20, and normalizing by the prior:
| ank14 | mfc20 | pca14 | |
| D-ALA | .2520 | .2157 | .2287 |
| D-centroid | .2862 | .2462 | .2457 |
Related to a suggestion of Dan's, I wanted to see if the score was correlated with the prior probability of the artist (i.e., popularity). So I made some plots, and it looks pretty much uncorrelated.
Posted by madadam at April 2, 2003 11:33 PM
