Ratings: Weighting is harming Prog Archives

Firstly, apologies, but I assume this has already been discussed to death, however I have to make my belated contribution to the discussion.

A bit of background about me first.

40 years old.  Been listening to Prog since about age 12 (Yes Album and Journey to the Centre of the Earth started it off.)   My collection just hit it's 500th Prog artist, covering artists from the 60's through to present day.  I consider myself a knowledgeable Prog fan.  What I don't consider myself is somebody capable of writing coherent or even interesting reviews for this site.

Since discovering this site it has become my main portal for prog information, new releases and filling gaps in my collection based on ratings and reviews.  And I have, according to my page, sumbitted 200 ratings.  I also buy lots of albums which are little known on the site and always make a point of voting to bring attention to them for the benefit of others.   So I'm doing my bit... or so I thought.

Yesterday I submitted a rating for Rocket Scientist - Revolution Road.  There were only 8 votes, mine was the 9th.  I checked the before and after rating as there was a 0% change, not even a 0.01 change.  So I did my research and discovered the weighting system that is currently in place and understood why my rating made no difference.

Which leads me on to my concern.

The weighting system currently in place means that ANY rating that contains a PA preferential member is biased towards that person's particular view (or those people).  In the case of the Rocket Scientists there are enough preferential members in those 8 previous votes that my vote may NO difference at all.  Which begs the first question, what's the point of having an open voting system if the weighting is so strongly working against those of us without writing skills or the time to do the writing?

This is actually more of a problem on those albums with few votes because it only takes 1 or 2 preferential members to vote to effectively lock down the rating to their bias.  And this means that, particularly if an album is not liked by a reviewer, that it will get lost amongst the higher rated albums, even if it's a geniune Prog gem.

So the first problem is that it's pointless to rate an album once a preferential member has done so.  The weighting makes the contribution pointless.   Which means it's actually pointless at all to rate albums on PA unless you're a preferential member.

But worse, this means that PA's charts only reflect the biases (and I'm not using that in a derogatory sense) of the preferntial members, and means the general audience are missing lots of potential gems.  Becuase as soon as a prefential member votes positively, or negatively, the album will appear/disappear from the charts.  (Yes, you can filter etc, but that doesn't solve the fundamental issue).  This for me, seriously undermines PA's usefulness as it is simply does not reflect the actual feelings of the entire user base.

What I want is a site that reflects the entire userbase, that allows those little known gems to become known by the wider audience and a site where one feels their contribution is useful.

Right now the weighting has fundamentaly broken PA's usefulness in that regard and should be addressed (removed weighting altogether, significantly lower the weights, or simply lock the rating to non-preferential users).  We would then all benefit from a more honest and open rating system rather than the current biased and pointless-to-contribute-to system that exists right now.

Thanks for listening!

Mark










Edited by Uncle Spooky - January 05 2009 at 05:27

Mark, your point would have merit if there were 3 or 4 sinister, caped Collabs rating the albums to the exlcusion of everyone else.  But the fact is this site has tons of "weighted" Collabs and adding many more all the time, meaning if you hang around for any length of time and *contribute* content to the site, you will soon be a weighted member and have your vote rewarded a little bit for your experience.  And yeah, it has pretty much been discussed to death, as you say.  It's a great system.  Feel free to go back and read other threads on the matter if you'd like more information on the reasons.  Thanks!

Angelo wrote: good to see you are so concerned with the site.

You're very welcome.  I suppose a bit more background could be useful; I was one of the founders of the Internet Movie Database and one of my responsibilities was managing the votes database including generation of results so I have a pretty good insight into what goes on and what is required.

However, if you do a bit of math here, you'll find that things are definitely not as extreme as you describe here.Looking at the Rocket Scientists album, you can see that it has 9 reviews and rating, including yours. 2 people voted 3 stars, the other 6 voted 4 stars. Now if I exclude your 4 stars and give all ratings equal weight, the 8 people before you would have awarded the album a 4.12. With your rating included, it would become 4.11 - a difference of 1/100 of a star, or about 0.25 percent.

The above is just one example.  But what if I had genuinely voted 1?  My vote would still have been lost to the weighting thus reinforcing my point that voting is pointless unless you are part of the PA in-crowd.

The weighted system results in a 3.86 - which is just as much a 4 star as a 4.11 would've been.

Well, you're actually talking about a 6% swing caused by weighting with this example.  For cash strapped prog fans trying to make purchasing decisions this could be significant.  But not only that you're not giving a true representation of the album compared to it's peers; you're giving a representation of the bias (again, used non-derogatorily) of the reviewer.  Sure, once you get more collaborators voting the results will tend to gravitate towards a true representative rating.  But it still doesn't stop non-preferential votes being useless.  And it doesn't address my concern about albums with, let's say just one negative rating from a preferential voter being lost to scrutiny from prog fans due to a poor rating.

Apart from that the weighting system does serve a purpose. We select collaborators and prog reviewers based on their knowledge and understanding of Prog.

But that selection process is clearly lacking as I haven't been approached ;)  I wonder how many other genuine, knowledgeable and honest voters slip the net?

At the same time anyone can add ratings and reviews. Even the biggest prog nitwit in the world can (and will) give 1 star or 5 stars based on the mood of the day. By applying the weighting system, the serious reviewing efforts of our collaborators balance out these extreme, and often unmotivated, ratings. In cases like Rocket Scientists' Revolution Road this doesn't show - because there's a lot of consensus on the album anyway, on others, like Selling England By The Pound, it does - because many people vote 1 or 5 stars there blindly, just because they hate Phil Collins or want to have Close To The Edge at the top of our albums chart.

Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out.

Over time, the weighting has been well thought through, and it has been redesigned quite a few times to become what it is now, taking into account just about any angle. I wouldn't worry too much about it if I were you - just do the math and you'll see how it works out.

What it works out to is that there is no point in anybody except preferntial voters voting.  So why not close the voting system completely?

On a completely different note: we have people who are not even native Englishs speakers who manage to squeeze out very useful reviews. Don't be afraid to describe how you feel about an album in your own words. Three sentences is sometimes enough, and it will increase the weight of your contribution.

I have contributed a review or two in the past but they were rejected.  Unfortunately I don't have the time to spend editing reviews to get them up to snuff.  I also offered to submit high quality scanned artwork, properly colour corrected from my collection but that didn't get anywhere.  Recently I made a new group submission, and that didn't get anywhere either.  There are some fundamental issues with this site.  Now that sounds like I'm bitching, but I'm not.  I'm just frustraed that PA could be so much better than it currently is and key to that is not setting up a division between those on the inside and those on the outside but by embracing everybody who wishes to make the site a better place.
 

I for one would be very curious to read why you gave only 1 start to Circa's 2007...

Taken from my review on LastFM:

"It's no secret that I am a Yes fan - Yes are without doubt the best band in the world ;) Always have been, always will be! :p So it's always with enthusiasm that I grab anything Yes-related. Unfortunately, its not always a happy experience and one Yes-related partnership which always disappoints me is anything with Billy Sherwood and yet I keep hoping - perhaps I should heed Yes's lyrics "Once bitten, twice shy"... This album is a compilation of all the worst aspects of prog - its pompousness, its excess, its lack of subtlety. Billy is like an amateur chef who has the recipe book in front of him but lacks any finesse in his creations and each meal leaves you gagging and trying to get the nasty taste out of your mouth. Perhaps it's a little unfair to point the finger so firmly in Billy's direction, but there is a body of work out there which suggests there is some justification... Sometimes, Billy, less is more. Less is more."

I stand by my rating on this one :D

With the amount of albums you rated personally, you could make Prog Reviewer by adding some motivation to your ratings - and as a reward get a peek on the inside of this site as well.

The problem there is that while I'd love my ratings to be useful, I don't want it to be because I'm a "chosen one".  The site as it stands is exclusionary.  I understand the arguments that weighting collaborators gives better results, but it doesn't actually stand up to scrutiny.  You can filter out junk voting transparently to ensure that only the good votes get through and you then end up with a more open and honest rating system.

How about this:  add a filter to the chart report pages to turn off weighting?  That would allow the "best" of both worlds...

Cheers,

Mark

Angelo wrote: Looking at the Rocket Scientists album, you can see that it has 9 reviews and rating, including yours. 2 people voted 3 stars, the other 6 voted 4 stars. Now if I exclude your 4 stars and give all ratings equal weight, the 8 people before you would have awarded the album a 4.12. With your rating included, it would become 4.11 - a difference of 1/100 of a star, or about 0.25 percent. The weighted system results in a 3.86 - which is just as much a 4 star as a 4.11 would've been.

I think your abacus needs rebeading - (3+3+4+4+4+4+4+4)=30 ... 30/8=3.75 and (30+4)/9=3.78

I think the others addressed your concerns more than adequately. However, I'd like to put my own two cents in. As Finn and Angelo already said, everyone can become a Prog Reviewer (if my memory serves me right, the requisites are illustrated in the Site Rules) provided their reviews show some quality. You are a native speaker of English (something I am not, yet I became a PR after three months here), and from what I can see you are very good at expressing yourself in writing - which would mean you could become a PR very quickly. 

Anyway, we are not "preferential" members, not "chosen ones", as you put it. We are just people who put in a lot of work on behalf of this site, and all for FREE. The higher weighting given to our reviews is a form of reward for all we are doing in order to make this site such an invaluable resource to lovers of prog rock. Do you think this is such a negative thing?

Mr ProgFreak wrote: UncleSpooky wrote: Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out. I'm curious - how can you easily weed out "erratic" 1/5 star votes? It's all open to opinion - if someone wants to assign 1 star to Close to the Edge or 5 stars to Love Beach, then I see no way to change these votes in a way that doesn't feel like censorship.

It's quite easy to spot lazy voters, whose votes, while valid, may actually be against the spirit of the vote, ie to generate accurate ratings for a given album.  You may then decide it's better to remove a percentage of the lazy votes to remove the "damage" they do.  Or you may choose to remove them all.  Or not.

But this is different to assigning weighting to collaborators.

If you look at IMDb you will see they breakdown votes and show IMDb staff as a completely separate rating and often the IMDb rating is different from the community: for example Dark Knight there is whole 10% difference.  If IMDb employed weighting the rank of Dark Knight would be very different...

This just shows that weighting the "staff" skews ratings according to their biases.

BTW: Saw your votes at Progfreak.com ... nice! My website really focuses on the ratings, while this one values reviews more highly. I may be slightly biased because I run PF ... but on the other hand I was a PA collaborator for many years. My suggestion: Use both websites - and others too! They each have their special advantages and disadvantages.

Heh, yeah I need to explore your site more.  I love the links to CDBaby, Amazon etc for available albums.  Makes purchasing very easy as long as I trust the votes :D

Mark


Edited by Uncle Spooky - January 05 2009 at 08:39

Uncle Spooky wrote: Mr ProgFreak wrote: UncleSpooky wrote: Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out. I'm curious - how can you easily weed out "erratic" 1/5 star votes? It's all open to opinion - if someone wants to assign 1 star to Close to the Edge or 5 stars to Love Beach, then I see no way to change these votes in a way that doesn't feel like censorship. It's quite easy to spot lazy voters, whose votes, while valid, may actually be against the spirit of the vote, ie to generate accurate ratings for a given album.  You may then decide it's better to remove a percentage of the lazy votes to remove the "damage" they do.  Or you may choose to remove them all.  Or not.

I see what you mean. Of course it's easy to spot those votes, meaning that when you see one you intuitively suspect abuse. But it's really difficult to implement automatic countermeasures, since there is no clear (mathematical) way to identify abusive votes.

At Progfreak.com I use a simple trick: Instead of simply computing the mean value the system also computes the median. The mean value of those two is then used as the result for the album. The median causes the value to "gravitate" towards the consensus. This means that if for example there are 9 5-star votes and only one 1-star vote, the result will be much closer to the 5-stars than with just using the mean:

mean: (45+1)/10 = 4.6
median: 5

result: (5 + 4.6)/2 = 4.8

UncleSpooky wrote: Heh, yeah I need to explore your site more.  I love the links to CDBaby, Amazon etc for available albums.  Makes purchasing very easy as long as I trust the votes :D Mark

Thanks! I'm glad that someone mentions those links ... it takes me quite some time to add them to the database. Each link to CDBaby, Amazon.com and eMusic.com is added manually.

And about trusting the votes: It depends all on whether you trust the voters ...

That's right Ivan.  The site has it's reasons and Mark and other are welcome to join in and collaborate in good spirit, or not.  He's dead wrong in his assertions though that our system is harming, we have a system that is the most successful anywhere in the promotion of quality music, we turn on new people every day to great prog.  And we even allow guys like Mark to speak out against the site publicly. 

Finnforest wrote: That's right Ivan.  The site has it's reasons and Mark and other are welcome to join in and collaborate in good spirit, or not.  He's dead wrong in his assertions though that our system is harming, we have a system that is the most successful anywhere in the promotion of quality music...

According to whom?

Mark

I'm sure this has already been mentioned, but it's also somewhat of a precautionary method from people who join the site only to sl*g a couple of records with 1 star review and give 5s to a couple others. I'm also sure that this has been mentioned, but this is one of the only 'non-Amazon' sites that allows ANYONE to post a review, if this were any other webzine you wouldn't even have that opportunity.

Should our opinions be worth more? The collaborators all started off in your same position and proved themselves as people who the website wants to represent opinions. I've always thought this way, so instread of bitching about it in my early days I simply worked to be the best so that my opinions would be valued. Now they are.

You say that the ratings aren't accurate? Please give an example of where they are not.