| Author |
 Topic Search  Topic Options
|
Queen By-Tor 
Special Collaborator
Honorary Collaborator
Joined: September 13 2006
Location: Xanadu
Status: Offline
Points: 16111
|
 Posted: April 10 2009 at 10:51 |
|
Just so you know... anyone who gives a rating without a review has the same weight. Collab or not, admin or not, M@X or not. To my understanding a rating without review is worth 1 mark in the overall construction of the marking - if anything there's just not enough reviews to support such a high mark so the average was easy to tip.
I also see that you have not reviewed or even rated that specific album.
|
 |
Desoc
Forum Senior Member
Joined: December 12 2006
Location: Oslo, Norway
Status: Offline
Points: 216
|
Posted: April 10 2009 at 11:25 |
 King By-Tor wrote:
Just so you know... anyone who gives a rating without a review has the same weight. Collab or not, admin or not, M@X or not. To my understanding a rating without review is worth 1 mark in the overall construction of the marking - if anything there's just not enough reviews to support such a high mark so the average was easy to tip.
|
Then you'll have to explain how come that particular album is rated as it is. Because if it is as you say then the maths don't add up, as far as I can see.
King By-Tor wrote:
I also see that you have not reviewed or even rated that specific album.
|
Heh, I expected that argument in some form. No, I don't own that album, in fact I haven't even heard it. I just try to follow what's moving around. In others words I don't have an opinion on who's rated right, I only pointed out the effect of one additional rating. However, I expect that what you're trying to say, is that I haven't rated or reviewed any albums. That argument has been used before in this thread: " Folks who have contributed little or nothing to the site come by and
tell Collabs, who have worked unbelievably hard over many long hours,
that what we are doing is wrong and worth little." I hope this is not how suggestions or critique is generally perceived in this forum - this collab is is certainly not describing my intention -, but pointing out this attitude was part of why I wrote the initial post. I plan to rate and review every single album I own at a point in time where I don't have two jobs parallelled with full-time studies and involvment in three different NGOs. I haven't found the peace to settle down and do that since I joined in 2006. I could have rated only, but that leaves out the possibility of reviewing later unless bugging an admin with several hundred deletion requests. If this makes my opinion less valued, pity. If this makes my engagement and thoughts in this forum irrelevant to anyone: Sad.
|
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout
Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
|
Posted: April 10 2009 at 11:29 |
^ I've just computed the numbers and that isn't true Mike - it does appear that collabs ratings-only are also weighted.
While I agree that collab weighting should not apply for rating-without-review (or at least be significantly less than it is at present), I would also add that rating-with-review is weighted even if you are not a collab. There will always be examples that prove how wrong the system is regardless of what the system is - an unweighted system is more vulnerable to abuse and will have more examples to illustrate that fact.
Allow me to use Pat Methany's The Way Up to illustrate this.
1. Without collab weighting it would have scored 4.27
2. Without collab weighting and without Igor's 1-star it would have scored 4.47
3. Without collab weighting and without both 1-stars it would have scored 4.69
4. Without collab weighing and with NO rating-only votes it would have scored 4.86
...therefore the two 1-star ratings have had a significant effect on the average, let's see that again with weighting:
5. With collab weighting it scores 4.07
6. With collab weighting, but without weighting Igor's 1-star rating-only it would have scored 4.67
7. With collab weighting, but without both 1-star rating-only votes it would have scored 4.84
8. With collab weighting and with NO rating-only votes it would have scored 4.91
the effect of weighting the collabs votes has evened-out the effects of unweighted 1-star votes, but as noted - a collab 1-star is a heavy bias, let's see whether there is a bias in the collab and non-collab opinion:
9. With collab weighting and only collab votes counted it would have scored 4.04
10. Without collab weighting and only collab votes counted it would have scored 3.66
11. With collab only reviews counted it would have scored 5.00
12 Without collab weighting and only review votes counted it would have scored 4.38
13 With ratings-only counted it would scored 3.91
1 collab and 1 non-collab both gave a poor rating, yet the reviews are generally more favourable towards the album than the ratings-only. The weighted average (4.07) is closer to the ratings-only average (3.91) [and I have no idea what that means]
So the question is - which of these numbers is a fair reflection of the album's worth?
The obvious answer is none of the above. As Mike by-Tor says - a fair reflection requires considerably more than 18 opinions, and without collab weighting the album would never have made the Top 100 anyway.
The average rating dropping like a stone was not caused by Igor's 1-star rating - it's elevated status of 4.67 was a direct result of the weighting, (without weighting it would have scored 4.47 before Igor's rating and 4.27 after), - what dragged down the average was all the 1, 3 & 4 star ratings-only votes.
|
|
What?
|
|
Queen By-Tor
Special Collaborator
Honorary Collaborator
Joined: September 13 2006
Location: Xanadu
Status: Offline
Points: 16111
|
Posted: April 10 2009 at 11:35 |
|
Thanks for that correction, Dean. I remember reading somewhere that a
collab rating without review would be the same strength as a non-collab
one... and I think that should be true anyways - Then again, you hardly
ever see a collab with such a differing opinion that doesn't at least
explain his views on the album.
But my overall view still stands, I don't know what's wrong with the site's staff having their opinion more pronounced. Not everyone agrees with the majority of reviews, but then again, when have you read a Rolling Stone article that you agreed with 100%?
Edited by King By-Tor - April 10 2009 at 11:36
|
|
Desoc
Forum Senior Member
Joined: December 12 2006
Location: Oslo, Norway
Status: Offline
Points: 216
|
Posted: April 10 2009 at 11:48 |
Dean wrote:
The average rating dropping like a stone was not caused by Igor's 1-star rating - it's elevated status of 4.67 was a direct result of the weighting, (without weighting it would have scored 4.47 before Igor's rating and 4.27 after), - what dragged down the average was all the 1, 3 & 4 star ratings-only votes.
|
Do Prog Reviewers count as much as those labeled Collaborator or Special Collaborator? I didn't know they were.
|
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout
Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
|
Posted: April 10 2009 at 12:02 |
|
^ yes - all Prog Reviewers are Collaborators
|
|
What?
|
|
Failcore
Forum Senior Member
Joined: October 27 2006
Status: Offline
Points: 4625
|
Posted: April 28 2009 at 17:00 |
|
Well I know you guys already know how I feel, but I just wanted to throw this out there. I know three people IRL (myself included) that don't write reviews for this site that often because of the weighting. That means there's probably even more out there. It just feels so thankless. Oh well, do what you want.
Edited by Deathrabbit - April 28 2009 at 17:02
|
|
Queen By-Tor
Special Collaborator
Honorary Collaborator
Joined: September 13 2006
Location: Xanadu
Status: Offline
Points: 16111
|
Posted: April 28 2009 at 17:09 |
Don't just try to guilt trip the site. Go out and earn your weighting... 
|
|
Finnforest
Special Collaborator
Honorary Collaborator
Joined: February 03 2007
Location: The Heartland
Status: Offline
Points: 16913
|
Posted: April 28 2009 at 17:23 |
|
If their decision to write about music they care about is based on a factor like this, then A) they don't care that much, and B) we're not missing much.
|
|
debrewguy
Special Collaborator
Honorary Collaborator
Joined: April 30 2007
Location: Canada
Status: Offline
Points: 3596
|
Posted: April 28 2009 at 21:50 |
|
Everybody hates me, nobody loves me , I'm gonna eat some wwwwoorrrmmmsss ...
|
|
"Here I am talking to some of the smartest people in the world and I didn't even notice,” Lieutenant Columbo, episode The Bye-Bye Sky-High I.Q. Murder Case.
|
|
Failcore
Forum Senior Member
Joined: October 27 2006
Status: Offline
Points: 4625
|
Posted: April 29 2009 at 00:05 |
Finnforest wrote:
If their decision to write about music they care about is based on a factor like this, then A) they don't care that much, and B) we're not missing much.
|
Wow, just forget I said anything. Just forget it. 
Edited by Deathrabbit - April 29 2009 at 00:05
|
|
russellk
Prog Reviewer
Joined: February 28 2005
Location: New Zealand
Status: Offline
Points: 782
|
Posted: April 29 2009 at 03:34 |
|
I write reviews because I want to share how the music made me feel. I don't care much for the ratings, because the number 1 or 2 or 3 or 4 or 5 can't tell you how I feel about an album. I gave 'In The Court of the Crimson King' 5 stars even though I don't like it, because I can objectively see why it is a masterpiece, but my review outlines the reasons why it doesn't work for me personally.
How can a rating without review do any of those things?
I applaud Uncle Spooky for his well-reasoned arguments, which I have just come across, but I can also see why ProgArchives does what it does. In the end, this is just another reason why I find myself draw to reviews to aid my purchasing decision rather than to ratings.
|
|
M@X
Forum & Site Admin Group
Co-founder, Admin & Webmaster
Joined: January 29 2004
Location: Canada
Status: Offline
Points: 4028
|
Posted: May 16 2009 at 13:04 |
Guys, what do you think of the new algo.
Here is some details about how we calculate the average rating of an album and the rank of an album.
Average rating: The classic calculation of the average but more weight
is affected to the rating of progarchives.com collaborators and to
rating with reviews.
- Rating only: Weight = 1
- Review by members : Weight = 5
- Review by PA Collaborators : Weight = 10 ?
|
|
Prog On !
|
|
Finnforest
Special Collaborator
Honorary Collaborator
Joined: February 03 2007
Location: The Heartland
Status: Offline
Points: 16913
|
Posted: May 16 2009 at 14:22 |
Max, it's great and your work much appreciated. The recent discoveries of manipulation by the man of a thousand Pink Floyd usernames is clear justification for weighting differences between ratings-only users, members, and Collabs. Hopefully this will put the topic to bed for awhile, where it needs a long peaceful slumber. Great job max 
|
|
crimson87
Prog Reviewer
Joined: January 03 2008
Location: Argentina
Status: Offline
Points: 1818
|
Posted: May 16 2009 at 16:12 |
The new weighting system is great. 5 for a normal user and 10 for a collab/PR is the way to go.
And "Brain Salad Surgery" is at the top!! I can't complain  |
|
Failcore
Forum Senior Member
Joined: October 27 2006
Status: Offline
Points: 4625
|
Posted: May 16 2009 at 22:18 |
|
Yeah it's much better than before. I have no complaints.
|
|
Desoc
Forum Senior Member
Joined: December 12 2006
Location: Oslo, Norway
Status: Offline
Points: 216
|
Posted: May 18 2009 at 13:24 |
|
This is a good change. Ideally, I'd like to see album ratings reflecting the opinion of
all contributors to this site, not mainly collabs (who should be rewarded for their much appreciated effort, but in other ways IMO), but this is definitely a step in the right direction.
And I have to say that I am impressed by M@x' ability to grasp the essence of ongoing debates and make corresponding changes to the benefit and liking of all. Even if the ideas and opinions come from newbies. Good job!
|
|
Donate monthly and keep PA fast-loading and ad-free forever.
/PAlogo_v2.gif "Progarchives.com Homepage")
Ratings: Weighting is harming Prog Archives