All:

 

As promised, here’s an update on the implementation of the new algorithm for the Top 50 most popular albums on the Home Page. Apologies for the delay but I’ve been busy at work. I am also including additional explanations and comments regarding the Top 50 debate.

 

This post is split into the following sections:

 

1. The purpose of the Top 50.
2. An explanation of the previous algorithms.
3. An explanation of the current algorithm.
4. My check on whether the current SQL query implements the current algorithm correctly.
5. Some notes on statistical analysis.
6. An explanation of why the previous algorithm was invalid.
7. Some additional comments.

 

The purpose of the Top 50

 

I proposed having a Top 10/20 to M@X in May last year. The title of my post was ‘List of the most popular albums overall’. My interest was popularity. Furthermore, as my post made clear, I am interested in popularity based on all ratings in the Archives, not just the Collaborators’ ratings (I’ll use the term Collaborator to refer to all official ProgArchives reviewers). I want to know popularity based on ratings by as many Prog fans as possible, the more the better.

 

It was always my intent that both the rating level and the number of ratings should be taken into account when assessing rank, i.e. that if Album A has, say, 50 ratings, all 4-star, and Album B has, say, 100 ratings, all 4-star, Album B would be ranked higher than Album A, i.e. more popular because more people like it, even though the typical rating of both is the same: 4 stars.

 

To me, in the case of the Archives album ratings, popularity is a function of both the number of people who like an album and the degree to which they like it. I am interested in a list of albums that takes into account both how much people like an album and also the number of people liking it. So I want to see an album that is rated highly by a large number of people appear higher up in a list than an album that is rated equally highly but by a smaller number of people.

 

Notice the words on the Home Page: ‘TOP 50: Prog Archives most popular titles’ (my emphasis).

 

By the way, the rating system was initially 1 to 5 stars but was changed in May last year to 0 to 5 stars for reasons that have nothing to do with the Top 50: someone wanted to be have an “it’s a bomb, don’t buy it” choice.

 

 

An explanation of the previous algorithms

 

Algorithm No. 1 summed all the ratings for each album in the Archives and ranked the albums according to the total number of stars per album (similar to integrating to find the area under a continuous distribution). Whilst this reflected the volume of ratings, it did not reflect the fact that, because of the definitions of each rating level, a low rating is an indication of unpopularity. Nevertheless, as low ratings have less impact on the sum of the ratings per album than high ratings, the net effect is not as bad as might be expected because the albums with higher ratings rise up a list ranked by total number of stars. But, clearly, a better algorithm was needed.

 

M@X later changed the algorithm to take into account only ratings by Collaborators. Thus was born Algorithm No. 2. I am not certain, but I believe it may have been a request by Easy Livin as he was concerned that fans of a particular album/band would encourage other fans of the album/band to visit the site and rate an album (let’s call it spamming, for want of a better word). Although I can understand his concern, personally I lost a lot of interest in the Top 20 (which was later increased to a Top 50) at that point because it now reflected only the ratings of a minority of the people rating and a minority of the overall number of ratings in the Archives. Another reason I found the Top 20 produced by Algorithm No. 2 less interesting is because I am also interested in knowing what is popular with the collective youth (teens and twenties) who listen to Prog Rock and its variants. The arithmetic mean age of Collaborators was, I believe, over 30 years old at the time – with plenty much older – and I believe it is still around that age now, although there are some very young Collaborators.

 

Algorithm No. 3 came about when I proposed in a forum poll that only Collaborators’ ratings accompanied by a review should be taken into account, my reasoning being that Collaborators, as ‘official reviewers’, should be expected to provide more than just a rating. Apart from this change the basic principle still applied, viz. the sum of the ratings per reviewed album was calculated and the totals sorted to arrive at a ranked list of albums.

 

Then the algorithm was changed again, to Algorithm No. 4. I’m not sure of the reason, although I believe it was probably because one of the Collaborators gave a low rating to a single album and it went up one position in the list (due, as I mentioned above, to the basic algorithm not reflecting that low ratings effectively signify unpopularity).

 

Algorithm No. 4 went back to using everybody’s ratings again, not just ratings made by Collaborators and having an accompanying review. The new algorithm ignored albums with less than 30 ratings and calculated the arithmetic mean of the ratings per album for the remainder, then created a ranked list of albums based on the arithmetic means. The difference in the order of the albums in the list was quite marked when compared to Algorithm No. 3. I found Algorithm No. 4 problematic because: a) the data is ordinal and therefore the arithmetic mean is an invalid way to assess rank; b) the distribution of ratings per album is generally not a Gaussian (Normal) Distribution, it is often skewed or bimodal or approximately uniform, all of which also make the use of an arithmetic mean meaningless (ouch!); c) the algorithm excluded albums with less than 30 ratings, when that is valid data; d) the resulting ranking did not take into account the number of ratings per album. Referring back to my original request to M@X, I wanted to see how popular albums are, so I wanted to see the ranking in a popularity list also reflect the number of people who rate an album highly (or lowly). Several other effects also bothered me (things like it being possible for two albums to swap order in the list if they both subsequently receive the same rating, for the arithmetic mean of albums to be the same when their distributions were not, etc.).

 

Someone posted recently in the Main Discussions forum, questioning why the JETHRO TULL album “Thick As A Brick” was at the top of the Top 50. He felt there are more deserving Prog albums and wondered whether there was a glitch in the algorithm. He received some unwarranted (in my opinion) criticism as a result. Although I love TAAB and indeed rated it as a 5-star album when I reviewed it, in no way do I think it is as popular to the collective Prog Rock fan base as a number of other albums. As I already knew that Algorithm No. 4 was invalid as a method of statistical analysis for this type of data, I thought it an appropriate time to think about an algorithm that reflects my original intention for the Top 50: a popularity list (a Prog album chart based on the Archives reviews), but that would also take into account that a rating of 2 stars and below is a vote of unpopularity. Thus was born Algorithm No. 5, the one currently generating the Top 50 list on the Home Page. This algorithm is explained below.

 

 

The current algorithm (Algorithm No. 5)

 

What I wanted, then, is an algorithm that takes into account the meaning of the star ratings, i.e. that 2 stars and below are in effect negative votes for an album and 3 stars and above are in effect positive votes for an album, and that also accommodates the fact that the total number of ratings for each album can differ. The algorithm is as follows.

 

For each album, perform the following calculation:

Sum the number of ratings per star-level and multiply it by the star-level less 2.5. Then sum the individual sub-totals and use the grand total to rank the album. For example, if Album A has two 5-star ratings and four 2-star ratings, and Album B has one 5-star rating and one 4-star rating, the grand total for Album A would be 2 x 2.5 + 4 x -0.5 = 3 and the grand total for Album B would be 1 x 2.5 + 1 x 1.5 = 4, and thus Album B would be ranked higher than Album A.

 

Consider the 6 rating levels:

1 -- 5 stars: Essential: a masterpiece of progressive music.

2 -- 4 stars: Excellent addition to any progressive music collection.

3 -- 3 stars: Good, but not essential.

4 -- 2 stars: Collectors/fans only.

5 -- 1 star:   Poor. Only for completists.

6 -- 0 stars: Bad. Do not buy!

By subtracting 2.5 from each star-level, the rating scale would become 2.5, 1.5, 0.5, -0.5, -1.5 and -2.5, which fits nicely with the meaning of the star ratings: a rating of 3 and above is a positive vote for an album, and a rating of 2 and below is a negative vote for an album, with the magnitude (i.e. absolute value) of the rating approximately signifying the strength of the vote for/against the album. Note that subtracting 2.5 and then taking the arithmetic mean for each album would make no difference to the rankings when compared to Algorithm No. 4 as it would just subtract 2.5 from every arithmetic mean, but using the grand total does make a difference.

 

Note that the subtraction has not weighted any of the data, nor have the data been normalised. ‘Weighting’ means multiplying by (or adding) a value to one or some of the original data to make that data have more effect on a calculation, i.e. to bias the calculation in someway. For example, multiplying the number of 5-star ratings by 2 would give the highest rating-level more weight than the other rating-levels. ‘Normalising’ means making the area under each frequency distribution of each album equal to the same value. Because the data is ordinal, the latter would not be a valid analysis technique in any case, as I explain later on.

 

Algorithm No. 4 excluded albums with less than 30 ratings. Algorithm No. 5 includes all albums and all ratings. I prefer not to exclude anybody’s rating just because the album rated happens to have a lower number of ratings than another. Bear in mind that the SQL query can generate a longer list (try entering the following URL in the Address box of your browser: http://www.progarchives.com/?MostPopular=150).

 

 

My check on whether the current SQL query implements Algorithm No. 5 correctly

 

M@X was able to provide me with the list of values calculated by the new SQL algorithm, plus a list of the total number of ratings for each of the 50 albums in the Top 50, but I had to derive the number of ratings per star-level for each album from the percentages displayed on each album’s page in the Archives. As those percentages are rounded, my calculation of the value using an Excel spreadsheet (let’s call this variable Popularity Rating, or PR for short) returned by the algorithm for each album differs very slightly from the PR returned by the SQL query, but basically confirms that the rank returned by the SQL query matches my algorithm.

 

In a forum thread last Saturday I expressed a doubt about the order of “Thick As A Brick” and “Tales From Topographic Oceans” in the Top 50 just after M@X implemented the new SQL query. However, the reason my calculations did not match the list on the Home Page was because I was incorrectly calculating the PV using only the ratings displayed with the reviews, and had forgotten that the Archives database also contains ratings without reviews. When taking all the ratings into account the SQL query returns a Top 50 list that matches my algorithm. So the SQL query does implement Algorithm No. 5 correctly.

 

By the way, the only change to the ratings of the albums in the Top 50 during the 12-hour period between M@X giving me the list of PVs calculated by the SQL query, and me transcribing the percentages for each of the star-levels for each of 50 albums, was due to a new 5-star review for the DREAM THEATER album “Awake”. I could get smug if it wasn’t for the fact that I know it could be put down to coincidence.

 

 

Some notes on statistical analysis

 

I’m not going to lecture you on the theory of statistical methods, but recommend that you at least read the explanations of some of the terms relevant to this discussion, given at the Web sites listed below.

 

Scales of measurement:

http://www.math.sfu.ca/~cschwarz/Stat-301/Handouts/node5.htm l

Note in particular the definitions of ordinal data and interval data.

 

Central tendency:

http://www.quickmba.com/stats/centralten/

 

Teaching statistics – Industrial realities not between the textbooks:

http://www-personal.umd.umich.edu/~williame/disai97.html

 

 

An explanation of why the previous algorithm (Algorithm No. 4) was invalid

 

The 6-level rating system used on the ProgArchives Web site produces ordinal data. Some might try to argue it is interval data because each level is precisely one star away from the adjacent level. However, it is not interval data for several reasons. Firstly, the stars are simply a numerical coding to indicate the rank of each rating. I’m sure you’ve seen this sort of thing before in market surveys:

 

“Rate the service you received. Enter one of the following values: 4 = Excellent, 3 = Good, 2 = Neither good nor bad, 1 = Bad, 0 = Very bad.”

 

“Pepsi tastes better than Coke. Enter one of the following values: 2 = Strongly agree, 1 = Agree, 0 = Neither agree nor disagree, -1 = Disagree, -2 = Disagree strongly.”

 

“TV is bad for children. Enter one of the following values: 4 = Strongly agree, 3 = Agree, 2 = Disagree, 1 = Disagree strongly.”

 

The above are ordinal data; the precise difference between each rating level (also called a scale point) cannot be quantified, unlike, for example, a measurement (50 cm, 40 cm, 30 cm, 20 cm, 10 cm). Ordinal data is qualitative. Measurements are quantitative. Arithmetic with ordinal data is meaningless. For example, if I rate an album as ‘Good but not essential’ (2 stars) and another album as ‘Excellent addition to any progressive music collection’ (4 stars), that does not necessarily mean that the second album is twice as good as the first.

 

Ordinal data should not be analysed using the methods devised to analyse parametric statistics, which assume the data is Normally Distributed (a symmetric, bell-shaped frequency distribution with specific characteristics). If you do, the resulting answers could be wrong. Non-parametric statistical methods should be used to analyse samples taken from a population with non-Normal distributions (skewed, bimodal, uniform, etc.). The arithmetic mean (a parametric statistical method) is certainly not a good estimate of the average in such distributions, and the median is often recommended instead.

 

Another reason why parametric statistical methods should not be used on the Archives data is because the spread in each sample (i.e. the set of each album’s ratings) must be similar in order to be able to compare samples. You just need to look at the rating frequency bar charts on each album’s page in the Archives to see that the rating distributions and variance (a meaningless statistic with non-symmetric distributions, anyway) are usually very different between albums.

 

Despite the possibility of erroneous conclusions, many people do perform arithmetic with ordinal data (including me – see earlier!) and use parametric statistical methods with ordinal data. In some cases the results are not too bad because the ordinal data approximates very roughly to interval data, but when the distributions are as non-Normal as those in the Archives, using parametric statistical methods, such as the arithmetic mean, are almost certain to result in errors.

 

The often-recommended measure of the average (typical value) of a sample is:

 

• the arithmetic mean for interval data if the distribution is Normal (i.e. not skewed, bimodal, uniform, etc.);
• the mode for nominal data;
• the median for ordinal data.

The album ratings are not only ordinal data, the sample distributions (the bar chart per album) are most definitely not usually Normal Distributions; they are often heavily skewed, or bimodal, or flattish (uniform), and some contain (valid) outliers. Meaningful analysis using arithmetic means and standard deviations requires quantitative data. The album ratings are not quantitative data; they are qualitative data. So the first mistake in Algorithm No. 4 was to use the arithmetic mean; the median might have been a better choice.

 

The second mistake in Algorithm No. 4 was to assume that the population (a term with a specific meaning in statistical methods) is the same for each album. In fact, each album rating is from a different population. I, for example, have never heard any of the COMUS albums. Perhaps that’s what Easy Livin was trying to say, in his post the other day, as an argument for using the arithmetic means to rank the albums. However, even if the ratings were interval data instead of ordinal data and were Normally distributed, using the arithmetic mean would be invalid precisely because the populations are different.

 

The third mistake in Algorithm No. 4 is due to the fact that the spread of the sample data (the ratings per album) is different. This also means that it is not possible to compare the samples using the arithmetic mean.

 

 

Some additional comments

 

The main bone of contention with the new algorithm seems to be that some DREAM THEATER albums have moved into the Top 50. As I write this, “Train Of Thought” is in second place in the list. Firstly, given the popularity of the band and some of its albums with Prog fans – particularly the younger ones – frankly I would be surprised not to see one or more of the band’s albums in a Prog Top 50 popularity list. Just look at the statistics at the time of writing:

 

227 ratings:

47% (107)       Essential: a masterpiece of progressive music

15% (34)         Excellent addition to any progressive music collection

13% (29)         Good, but not essential

9% (20)           Collectors/fans only

15% (34)         Poor. Only for completists

2% (4)               Bad. Don’t buy!

 

(Values don’t quite correspond due to rounding.)

 

Now look at the PINK FLOYD album “Dark Side Of The Moon”, immediately below TOT in the Top 50 as I write this:

 

117 ratings:

80% (94)         Essential: a masterpiece of progressive music

9% (11)           Excellent addition to any progressive music collection

7% (8)               Good, but not essential

2% (2)               Collectors/fans only

2% (2)               Poor. Only for completists

1% (1)               Bad. Don’t buy!

 

(Values don’t quite correspond due to rounding.)

 

Clearly a high percentage of those people who rated DSOTM rated it as a masterpiece, and a lower percentage of those people who rated TOT rated it as a masterpiece. But in absolute terms TOT has more Masterpiece ratings than DSOTM, more Excellent ratings than DSOTM, and, taking into account the low ratings (subtracting low ratings from the total), the net effect is still that TOT is just above DSOTM in the TOP 50 popularity chart (not by much though: three more ratings could change that, they’re that close).

 

You can see how popular TOT is from the chart below, which shows the number of people who have rated TOT and DSOTM at each of the 6 star-levels. The number of people who approved of the albums is shown above the x-axis and the number of people who disapproved of the album is shown below the x-axis. The ‘Poor. Only for completists’ vote cancels out the entire ‘Excellent addition’ vote, and the ‘Collectors/fans only’ vote cancels out about two thirds of the ‘Good, but not essential’ vote. Tot it all up and the popularity of the two albums in absolute terms (i.e. in terms of the net number of people who rate the album highly) is close.

 

The only mystery to me is why DREAM THEATER fans have not been lobbying loudly in the forums for the band to get more recognition on the site for its music and achievements: its albums are very popular. If I were a DT fan I think I would feel quite aggrieved at the way some forum members dismiss the band.

 

I just picked DREAM THEATER as an example. I am not a fan of the band, but the popular DT albums were a glaring omission from the Top 50 popularity chart created using the previous algorithm. Looking at the list today, I think it is quite reasonable. I’ve just now seen GENESIS – “Selling England By The Pound” leapfrog “Train Of Thought”, too. I am reasonably happy with the new algorithm. I won’t pretend it’s perfect, but empirically it seems to reflect the popularity of albums with the ProgArchives fan base at least, and takes into account the meaning of the ordinal ratings: an approving rating increases an album’s Popularity Value and a disapproving rating decreases an album’s Popularity Value. If the PVs of albums in the Top 50 are close enough, this could change an album’s position in the list. And now the album will move the correct way.

 

And just so that people know I know (!), I do realise that my algorithm has converted the ordinal data into interval data. But the net effect is closer to my goal of producing a Top 50 that attempts to show popularity taking into account quality and quantity.

 

I’d like to thank richardh. He pragmatically pointed out in a succinct way the flaw in the previous algorithm that had been bugging me for some time, and rekindled my interest. Iván tried to counter Richard’s argument using the (inappropriate) arithmetic mean, but Richard had correctly identified the silliness of some of the rankings in the previous Top 50. And, although he first poured scorn on DREAM THEATER’s entry into the Top 10, his subsequent post in the forum thread ‘The List’ is a paragon of common sense:

 

“The list is actually reasonably well balanced. It will never please everyone. Yes - Close To The Edge is NO1 which seems ok to me. The fact that Dream Theatre are very popular is not a crime and doesn't mean they are not worthy of having a high place in the charts. Maybe we should just accept this. They do qualify as a prog band I believe.”

 

I could almost (but not quite!) forgive him for loathing TRIUMVIRAT! Cheers, Richard

 

I’d also like to thank M@X for graciously indulging me as usual, and for the speed with which he responded. He both wrote and tested the new SQL query very quickly. By the way, the new SQL code is simpler and faster than the previous query, which is not surprising when you consider that the previous query checked every album to see if it had 30 or more ratings and, additionally, performed a floating-point division for each album. The new algorithm does none of these.

 

To those of you who complain that the site should not have a Top 50 list at all, or that it should be confined to an elite who somehow know what is best for the rest of the fans: firstly, the Top 50 is just a bit of fun – something to scan and discuss – and secondly I like being able to see what albums are popular with the Prog Rock fan base. If Easy Livin’s concern about spammers proves correct then we’ll have to go back to limiting the Top 50 to Collaborators ratings only, but let’s see how things go.

 

Also, in my view we should try to avoid elitism when it comes to Prog bands/albums; quality is subjective in the case of the arts (beauty is in the eye/ear of the beholder). Just because I don’t like a particular album does not mean that album contains poor quality music, especially as there is no means of measuring the quality of music; it’s not like being able to define a quality parameter such as number of threads per square centimetre of cloth, amount of gold in an alloy, number of currents in a bun, thickness of a coat of paint, etc.). One man’s meat is another man’s poison: an album that I think contains top quality music is a coaster to richardh. It may be hard to swallow for some, but sometimes popularity might have to be used as a rough guide to quality if there is no other way of determining it. Some people can’t stand the music of ELP and YES, but those two bands alone have sold tens of millions of albums. Seems like quite a few people thought those albums were good.

 

It’s been fun over the last few days thinking about the Top 50 and ways of deriving it.

 

Best Regards to y’all.

 

P.S. If anyone mentions the word ‘average’ again I won’t be held responsible for my actions

 

Edited by Fitzcarraldo

What you've done is.

Calculating the average rating, substract 2,5 points and multiply by the number of ratings
(nothing more and nothing less)

The outcome is hugely dependent on the number of ratings, as long as the average is above 2,5 the album would come op on top of the list, as long as there's enough ratings.

example,

an album has 3 * 5 star ratings and 3 * 4 star ratings. the outcome would be 27 points

another album has 0 * 5 stars, 0 * 4 stars, 70 * 3 star ratings and 10 * 2 star ratings the outcome would be 30 points, making the latter more popular on this list, solely because the number of voters from the second album outweigh the number of voters of the first album.

All that's been measured now is how many times has an album been rated. Of course the average is still of importance this way, but the outcome is highly polluted.

With this system we measure how popular a band is on a given time (popular/much bought album from last year was obviously Train Of Thought, consequently it has been rewarded with a vast number of reviews, average rating 3,65. that is ok, but it's not the average a timeless masterpiece will get (CTTE 4,80. DSOTM 4,61)

Where measuring popularity in a fixed time on a continuing timescale (popularity now, will still be measured in 30 years, so after the album is long forgotten the album will still be in the top 50, because a number of persons decided to rate the album 3 stars (meaning not bad/reasonably good)

Still.

 

I do agree that using a strict average is not perfect aswell, and the number of reviews/ratings does signify something, but this isn't a good algorithm.

Some good points in it nonetheless.
Substracting 2,5 points from the average is a good thing (without it an album averaging 2,4 with 100 reviews will come higher than an album averaging 4 with 10 reviews, while in this system it comes in very low below )

If you insist on making the number of ratings significant in the calculation may I suggest you top the number of ratings at 100. So if an album has over 100 ratings, the factor isn't the exact number of ratings but 100. above the 100 ratings the average takes over in importance.

I can't take the list as it is now seriously, for it is hugely flawed.

 

 

Edited by tuxon

Thanks for all the background info Fitz, you've put a lot of tiem and thought (or is that train of thought) into it.

I'll not mention averages for fear of a visit on a dark night.

As you say, any chart should be taken with a pinch of salt, and will not be perfect.

The overriding concern I have about the present one remains though, and I think Tuxon's post implies a similar worry. That is, a mediocre album can ride high in the chart, simply because a lot of peope have reviewed it, and said they "quite like it." Anything "Good" or above results in additonal "points" being gained. This means that albums which are universially acclaimed as masterpieces can appear lower in the chart than quite good albums, just because more people have rated the quite good ones. I don't like that, that to me does not fit well with the word popular.

There do seem to be a disproportionate number of reviews and ratings already for TOT on the site when compared to amount of discussion there is about Dream Theater in the forum. Has there perhaps been some "spamming" already? Perhaps a limit on the maximum number of reviews, say the first 100 would help?

Another thought. In the olympics, the medals table takes account of all the gold medals gained first. One gold medal is worth more than any number of silver. Worth condsidering?

Unless it's the sychronised diving!

tuxon wrote:   In your calculation Train Of Thought acquires 262 points   107 5 267,5 34 4 51 29 3 14,5 20 2 -10 34 1 -51 4 0 -10 262 An album with 100 solely 5 star ratings would have less points than Train Of Thought. Please explain the logic in this

tuxon,

 

You can see better how popular the two albums are from the chart below, which shows the number of people who have rated TOT and TUXON (your hypothetical album with only 100 5-star ratings) at each of the 6 scale points (star levels).

The number of people who approved of the albums is shown above the x-axis and the number of people who disapproved of the album is shown below the x-axis. I have represented the chart bars this way to indicate the opposing nature of the ordinal data (approval/disapproval) and also because that's how Algorithm No. 5 proceeds.

Firstly, remember that the scale on the x-axis is ordinal and represents attitudes which, very roughly speaking could be considered as ranked 'sublime', 'excellent', 'good', 'not good', 'awful', 'use it for clay pigeon shooting'. They are, very roughly speaking, symmetrical about the line between 'good' and 'not good' on the x-axis.

For TOT, the ‘Poor. Only for completists’ vote cancels out the entire ‘Excellent addition’ vote, and the ‘Collectors/fans only’ vote cancels out about two thirds of the ‘Good, but not essential’ vote. Tot it all up and the popularity of the two albums in absolute terms (i.e. in terms of the net number of people who rate the album highly) is close, with TOT just slightly more popular. Hopefully it is easier to understand when presented in pictorial form. You have to forget about arithmetic means (well, not forget them completely, but understand that the arithmetic mean is not always the appropriate tool for making a ranking decision - see also my previous post).

Easy Livin wrote: "Thanks for all the background info Fitz, you've put a lot of tiem and thought (or is that train of thought) into it."

Easy Livin wrote: "The overriding concern I have about the present one remains though, and I think Tuxon's post implies a similar worry. That is, a mediocre album can ride high in the chart, simply because a lot of peope have reviewed it, and said they "quite like it." Anything "Good" or above results in additonal "points" being gained. This means that albums which are universially acclaimed as masterpieces can appear lower in the chart than quite good albums, just because more people have rated the quite good ones. I don't like that, that to me does not fit well with the word popular."

I'll tackle this one at the same time as I reply to tuxon's post, Easy, if that's OK. I'm quite busy at work at the moment, so have less time to play. Actually, I wrote most of my looong post at 10,000 metres.

Easy Livin wrote: "There do seem to be a disproportionate number of reviews and ratings already for TOT on the site when compared to amount of discussion there is about Dream Theater in the forum. Has there perhaps been some "spamming" already? Perhaps a limit on the maximum number of reviews, say the first 100 would help?"

If you look at the reviews page for TOT, the rate of posting looks quite normal. It could simply be because the album is popular. I can't tell about the ratings posted without reviews, of course, but maybe Max would be able to see a date_time stamp in the database for ratings only, as well as reviews? The band just seems to be very popular - I see their CDs in many shops. Perhaps DT fans don't find ProgArchives a very welcoming place to discuss Metal? Do our forums tend to focus on the 'classic bands' more? Perhaps the DT fans prefer to frequesnt other sites. I have browsed a few DT sites in the past to find out more about the albums, and they seemed to be thriving. I got the impression that the band is very popular.

Easy Livin wrote: "Another thought. In the olympics, the medals table takes account of all the gold medals gained first. One gold medal is worth more than any number of silver. Worth condsidering?"

Do you mean apply a weighting factor to 5-star ratings, Easy? I'd prefe to keep weighting factors in the back pocket for now. What is niggling me about the data at the moment is that the distributions are very varied, heavily skewed in many cases, the populations different, and the ordinal scale (the words, i.e. the meaning of the scale points) not necessarily of equal interval, which screws up any parametric statistical analysis even more. One thing that tuxon, bless him, could do - 'cause I don't have the time - is to look at the medians for all the distributions and see if that can be used as a more meaningful ranking measure. There are ways to deal with skew: geometric means and all sorts of heavy non-paramentic stats methods but they involve some heavy number crunching so that rules them out.

And so to bed, as Samuel said.

Fitz, and all interested parties,

Well I've interpret and calculated the data from the albums in the top 50.

I tried to copy-paste the outcome, apparently the post couldn't handle it, so I'll give a small summary of my findings.

 

Number of ratings
Average number of ratings of the albums currently in the top 50 = 78
Number of ratings for Train Of Thought = 229 (3*average)
number two on the scale of ratings is Yes CTTE, with 145 (86 less than TOT)

Effect on the number of ratings on the Top 50
All top 10 albums have 100 or more ratings
The lowest 100 review album is Yes TFTO ranking 17

Averages
The lowest average is from Dream Theaters Train of Thought, well below Pink Floyds the Division Bell. These are also the only two averaging below 4 stars.

My recomandation for a failsave mechanism to be incorporated with the algorithm stands. this means an IF-formula in the calculation.
Formula : (IF(number of ratings>100;100;number of ratings)*(average rating)

It isn't really nescesary to substract 2,5 from the average, the results remain the same

This way the number of ratings stay very important (only 12 albums currently have 100 ratings or more, and only 25 albums have more than 70 ratings) I doubt there are albums with more than 80 reviews outside the current top 50, and if so their averages are below 3,00

I tried BebieM's proposal aswel, with different ways, the results appeared normal, but there is a fault in it (album with 100 reviews, 50 * 4 stars and 50 3 stars would end up with 4 stars as average), The idea 'behind it was not bad in itself BTW.

 

Concluding:

The number of reviews are a good indication of popularity and ranking, but they are not a good indication of quality.
The algorithm I provided is better. The number of ratings setting at 100 is however arbitrary (the former algorithm had a similar formula, using 30 (approximation of that algorithm, not being specific), I believe that was something like this. If(number of ratings<30;Average;0). it didn't had the problem of hugh quantities of reviews, but it did have the problem that relatively unknown bands were easaly incorperated in the top 50 (volatility of the mean, meaning 1 * 5 star rating weighs relatively heavy on the mean).

with the proposed algorithm, high quantities of reviews remain of significant influence, but an overdose on reviews does not poison the outcome. Like I said the choice for 100 is arbitrary, 50 or 150 are also possible. The higher you make it, the more impact the amount of reviews have.

Any Questions ?

 

WOW! I've no intention of reading all that but  all the same time.

First. I think the list should be and stay at the home page. I think a lot of people are interested in these kinds of lists. i think especially to new-comers to this site it's a fast way to know which are the major bands and albums in progressive rock.

second I've been coming around to your vision, and I agree that numbers of ratings are a very good measuring tool for populatity.

i've been working on some other possible algorithms, one of them I think could be interesting.

algorithm number 6

almost the same as the current algorithm, but punishing more severly for low ratings.

album score = (average rating - 2,5) * (number of ratings - 0,5*numbers of 3 star ratings - number of 2 star ratings, - number of 1 star ratings - number of 0 star ratings)

I like the results from this calculation It prevents albums with mostly 3 star ratings from climbing to a top position (with enough ratings it is possible, but then your argument that it has to be a well-known, popular album provides reason for their inclusion in the list.)

and it tackles the problem of my suggested 100 ratings limit as maximum.

Fitz,

A very interesting and detailed post. You have given this far more thought than I have, and I bow to your considered recommendations. The proof of the pudding as they say is in the chart which appears on the home page, and I'm happy enough that overall it contains the right albums.

I think you and I will aways differ on how important popular vs best is. I still feel any chart on the home page should as far as possible list the cream of prog. It is always said that the singles charts don't list the best music, but the most popular, and to some extent that is what our current chart does.

However, I think for the time being, we have taken things as far as is necessary, and should let the current chart bed in.

Till the next time...