Progarchives.com has always (since 2002) relied on banners ads to cover web hosting fees and all.
Please consider supporting us by giving monthly PayPal donations and help keep PA fast-loading and ad-free forever.
/PAlogo_v2.gif "Progarchives.com Homepage") |
|
Post Reply 
|
Page 123 6> |
| Author | |||
Desoc 
Forum Senior Member 
Joined: December 12 2006 Location: Oslo, Norway Status: Offline Points: 216 |
 Topic: Ratings: Weighting is harming Prog ArchivesPosted: May 18 2009 at 13:24 |
||
|
This is a good change. Ideally, I'd like to see album ratings reflecting the opinion of
all contributors to this site, not mainly collabs (who should be rewarded for their much appreciated effort, but in other ways IMO), but this is definitely a step in the right direction.
And I have to say that I am impressed by M@x' ability to grasp the essence of ongoing debates and make corresponding changes to the benefit and liking of all. Even if the ideas and opinions come from newbies. Good job! |
|||
 |
|||
|
Failcore
Forum Senior Member
Joined: October 27 2006 Status: Offline Points: 4625 |
Posted: May 16 2009 at 22:18 |
||
|
Yeah it's much better than before. I have no complaints.
|
|||
|
|||
|
crimson87
Prog Reviewer 
Joined: January 03 2008 Location: Argentina Status: Offline Points: 1818 |
Posted: May 16 2009 at 16:12 |
||
|
The new weighting system is great. 5 for a normal user and 10 for a collab/PR is the way to go.
And "Brain Salad Surgery" is at the top!! I can't complain
 |
|||
|
|||
|
Finnforest
Special Collaborator 
Honorary Collaborator Joined: February 03 2007 Location: The Heartland Status: Offline Points: 16913 |
Posted: May 16 2009 at 14:22 |
||
|
Max, it's great and your work much appreciated. The recent discoveries of manipulation by the man of a thousand Pink Floyd usernames is clear justification for weighting differences between ratings-only users, members, and Collabs. Hopefully this will put the topic to bed for awhile, where it needs a long peaceful slumber.
Great job max  |
|||
|
|||
|
M@X
Forum & Site Admin Group
Co-founder, Admin & Webmaster Joined: January 29 2004 Location: Canada Status: Offline Points: 4028 |
Posted: May 16 2009 at 13:04 |
||
|
Guys,
what do you think of the new algo.
Here is some details about how we calculate the average rating of an album and the rank of an album. - Rating only: Weight = 1 - Review by members : Weight = 5 - Review by PA Collaborators : Weight = 10 ? |
|||
|
Prog On !
|
|||
|
|||
|
russellk
Prog Reviewer
Joined: February 28 2005 Location: New Zealand Status: Offline Points: 782 |
Posted: April 29 2009 at 03:34 |
||
|
I write reviews because I want to share how the music made me feel. I don't care much for the ratings, because the number 1 or 2 or 3 or 4 or 5 can't tell you how I feel about an album. I gave 'In The Court of the Crimson King' 5 stars even though I don't like it, because I can objectively see why it is a masterpiece, but my review outlines the reasons why it doesn't work for me personally.
How can a rating without review do any of those things? I applaud Uncle Spooky for his well-reasoned arguments, which I have just come across, but I can also see why ProgArchives does what it does. In the end, this is just another reason why I find myself draw to reviews to aid my purchasing decision rather than to ratings. |
|||
|
|||
|
Failcore
Forum Senior Member
Joined: October 27 2006 Status: Offline Points: 4625 |
Posted: April 29 2009 at 00:05 |
||
Wow, just forget I said anything. Just forget it.  Edited by Deathrabbit - April 29 2009 at 00:05 |
|||
|
|||
|
debrewguy
Special Collaborator
Honorary Collaborator Joined: April 30 2007 Location: Canada Status: Offline Points: 3596 |
Posted: April 28 2009 at 21:50 |
||
|
Everybody hates me, nobody loves me , I'm gonna eat some wwwwoorrrmmmsss ...
|
|||
|
"Here I am talking to some of the smartest people in the world and I didn't even notice,” Lieutenant Columbo, episode The Bye-Bye Sky-High I.Q. Murder Case.
|
|||
|
|||
|
Finnforest
Special Collaborator
Honorary Collaborator Joined: February 03 2007 Location: The Heartland Status: Offline Points: 16913 |
Posted: April 28 2009 at 17:23 |
||
|
If their decision to write about music they care about is based on a factor like this, then A) they don't care that much, and B) we're not missing much.
|
|||
|
|||
|
Queen By-Tor
Special Collaborator
Honorary Collaborator Joined: September 13 2006 Location: Xanadu Status: Offline Points: 16111 |
Posted: April 28 2009 at 17:09 |
||
|
Don't just try to guilt trip the site. Go out and earn your weighting...
|
|||
|
|||
|
Failcore
Forum Senior Member
Joined: October 27 2006 Status: Offline Points: 4625 |
Posted: April 28 2009 at 17:00 |
||
|
Well I know you guys already know how I feel, but I just wanted to throw this out there. I know three people IRL (myself included) that don't write reviews for this site that often because of the weighting. That means there's probably even more out there. It just feels so thankless. Oh well, do what you want.
Edited by Deathrabbit - April 28 2009 at 17:02 |
|||
|
|||
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575 |
Posted: April 10 2009 at 12:02 |
||
|
^ yes - all Prog Reviewers are Collaborators
|
|||
|
What?
|
|||
|
|||
|
Desoc
Forum Senior Member
Joined: December 12 2006 Location: Oslo, Norway Status: Offline Points: 216 |
Posted: April 10 2009 at 11:48 |
||
Do Prog Reviewers count as much as those labeled Collaborator or Special Collaborator? I didn't know they were. |
|||
|
|||
|
Queen By-Tor
Special Collaborator
Honorary Collaborator Joined: September 13 2006 Location: Xanadu Status: Offline Points: 16111 |
Posted: April 10 2009 at 11:35 |
||
|
Thanks for that correction, Dean. I remember reading somewhere that a
collab rating without review would be the same strength as a non-collab
one... and I think that should be true anyways - Then again, you hardly
ever see a collab with such a differing opinion that doesn't at least
explain his views on the album.
But my overall view still stands, I don't know what's wrong with the site's staff having their opinion more pronounced. Not everyone agrees with the majority of reviews, but then again, when have you read a Rolling Stone article that you agreed with 100%? Edited by King By-Tor - April 10 2009 at 11:36 |
|||
|
|||
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575 |
Posted: April 10 2009 at 11:29 |
||
|
^ I've just computed the numbers and that isn't true Mike - it does appear that collabs ratings-only are also weighted.
1. Without collab weighting it would have scored 4.27 ...therefore the two 1-star ratings have had a significant effect on the average, let's see that again with weighting: the effect of weighting the collabs votes has evened-out the effects of unweighted 1-star votes, but as noted - a collab 1-star is a heavy bias, let's see whether there is a bias in the collab and non-collab opinion: So the question is - which of these numbers is a fair reflection of the album's worth? |
|||
|
What?
|
|||
|
|||
|
Desoc
Forum Senior Member
Joined: December 12 2006 Location: Oslo, Norway Status: Offline Points: 216 |
Posted: April 10 2009 at 11:25 |
||
Then you'll have to explain how come that particular album is rated as it is. Because if it is as you say then the maths don't add up, as far as I can see.
Heh, I expected that argument in some form. No, I don't own that album, in fact I haven't even heard it. I just try to follow what's moving around. In others words I don't have an opinion on who's rated right, I only pointed out the effect of one additional rating. However, I expect that what you're trying to say, is that I haven't rated or reviewed any albums. That argument has been used before in this thread: "Folks who have contributed little or nothing to the site come by and tell Collabs, who have worked unbelievably hard over many long hours, that what we are doing is wrong and worth little." I hope this is not how suggestions or critique is generally perceived in this forum - this collab is is certainly not describing my intention -, but pointing out this attitude was part of why I wrote the initial post. I plan to rate and review every single album I own at a point in time where I don't have two jobs parallelled with full-time studies and involvment in three different NGOs. I haven't found the peace to settle down and do that since I joined in 2006. I could have rated only, but that leaves out the possibility of reviewing later unless bugging an admin with several hundred deletion requests. If this makes my opinion less valued, pity. If this makes my engagement and thoughts in this forum irrelevant to anyone: Sad. |
|||
|
|||
|
Queen By-Tor
Special Collaborator
Honorary Collaborator Joined: September 13 2006 Location: Xanadu Status: Offline Points: 16111 |
Posted: April 10 2009 at 10:51 |
||
|
Just so you know... anyone who gives a rating without a review has the same weight. Collab or not, admin or not, M@X or not. To my understanding a rating without review is worth 1 mark in the overall construction of the marking - if anything there's just not enough reviews to support such a high mark so the average was easy to tip.
I also see that you have not reviewed or even rated that specific album. |
|||
|
|||
|
Desoc
Forum Senior Member
Joined: December 12 2006 Location: Oslo, Norway Status: Offline Points: 216 |
Posted: April 10 2009 at 09:20 |
||
|
Well, I realize that this thread has been inactive for some weeks now, but I feel the need to make the question reappear, partly because I feel the debate was largely inconclusive.
I didn't join the crowd the last time around, but I must admit that the debate puzzled me. I have the deepest respect for most of the collaborators and the time and effort they put into this site. But this thread was a curious showcase.
Regarding the debate
To my eyes, the debate consisted mainly of non-collaborators (in particular, but not limited to one single person) that was questioning a particular (and very visible and impactable) feature of the site, against a massive load of collaborators who (with a couple of exceptions) went right down in the trenches to defend their privileges. I don't think these privileges are the reason they are active, so the total reaction was peculiar, and it certainly stopped me from engaging in the debate.
Well, this is not meant to be a rant against collabs, whose efforts - as I said - I admire. But this thread leaves the impression that there is a certain defiance against the common people here, which is an impression that gains noone, regardless of its accuracy. (Something similar can be found in this thread: http://www.progarchives.com/forum/forum_posts.asp?TID=55758 and this: http://www.progarchives.com/forum/forum_posts.asp?TID=55741&PN=2) I think Mark had valid points, and I was surprised at how he was met. Take it as friendly advice.
Reviews vs ratings
I believe that collabs in general write better reviews than non-collabs. Thus, I think the exhibition of their reviews should reflect this. For my part, the frontpage feed could consist of collab reviews only. And collabs should be rewarded manyfold for their efforts in various ways.
But ratings are an entirely other issue. Being a good reviewer doesn't mean that your opinion is more qualified. And what is the point of the rating system? First and foremost it is to show the standing of an album amongst the community at large. As such, the current system must be said to be misleading.
Possible changes?
When I say that the debate was largely inconclusive, I refer to the fact that most of the defendants where people who "gained" on the current system, and those few who raised voices were (with a couple of exceptions) not. But there were a few concrete proposals that hardly anyone commented.
I'm curious to know if displaying different averages is something that could be considered, or alternatively why not. Even if the current system remains the "standard", would anything be lost if people could additionally chose basic average, non-weighted ratings, collab-only ratings etc?
And what about having a filter on the chart pages that removes the weighting? Or is that technically impossible?
A specific example
I'd like to close with an example of how the current system works. Let me attract your attention to this album: http://www.progarchives.com/album-reviews.asp?id=12217
This album had a rating of 4,67 or something and was on the top 100 chart if minimum # of ratings was lowered. Then one collab gave it a 1-star rating, without review, and it dropped like a rock. First of all, this shows that it's dead wrong that it's the quality of reviews that matters. Secondly, it shows that this system doesn't primarily reward collabs - it primarily punishes the other 17 raters, many of them with well-crafted reviews. And this would have been the case also if the collab in question wrote a review alongside.
So in conclusion, if the site owners feel that weighting by mixing reviews and ratings is important for giving incentives to writing reviews, then I will hold my peace. But then my advice as a regular user would be to at least treat all reviews equal, and rather give other kinds of bonuses to collabs. At any rate, the current weighting is - sorry - ridiculously biased.
|
|||
|
|||
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575 |
Posted: January 10 2009 at 07:15 |
||
Doubly so if those two people have similar tastes to you.
The question then is which way to tweak the average. Do you tweak it in favour of the "lovers" or "haters"? Common sense says towards the "lovers" ... (a low rating by a "hater" is in effect a high rating
) ... but the problem there is what if the low ratings were from people who love the genre/artist but hate the album.Standard deviation does give more information - we could flood the page with numbers, but that is a distraction which would open us to even more criticism by people who would not appreicate what the numbers mean. We do plot the distributions on each album page - people should be using that graph to draw their own conclusions rather than concentrating on the individual scores (sorry they don't display properly here, but in essence the 3.74 rating is better explained by the 44% of people who gave the album 4-stars):
3.74
(97 ratings)
|
|||
|
What?
|
|||
|
|||
|
Mr ProgFreak
Forum Senior Member
Joined: November 08 2008 Location: Sweden Status: Offline Points: 5195 |
Posted: January 10 2009 at 06:35 |
||
It would be interesting for the users to see the bayesian average along with the arithmetic mean, but from my own website I can say that it's a bit difficult to implement. However I'll try to do that.
Well, I think that the numbers are quite useful. Of course they don't represent the "true" rating of the album ... there is no such thing. As far as I'm concerned, ratings are useful because they enable the system to provide suggestions - even if only two people rate something highly, I might want to check it out. BTW: I already thought of what you're describing in the highlighted section. At PF I'm calculating the standard deviation for each album, and here you can see the album with the highest values. For large numbers of ratings with about equal "haters" and "lovers", it might even make sense to tweak the resulting average in some way. At PF I'm doing that by also considering the median value in the resulting average. |
|||
|
|||
|
Post Reply
|
Page 123 6> |
| Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
Ratings: Weighting is harming Prog Archives
Topic Options
Finnforest wrote:
Essential: a masterpiece of progressive music (22%)
Excellent addition to any prog music collection (44%)
Good, but non-essential (21%)
Collectors/fans only (10%)
Poor. Only for completionists (3%)
- I originally used the term Bayesian Algorithm, which can apply to either filtering or weighting, however since I said it was used to calculate the Top 100 it infers weighting.