Page 4 of 24

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 1:05 am
by Bubba Suess
Bob Burd wrote:Regarding comments about an "abstain" option.
I figured not voting at all would be the same (score-wise, it would), but I can see that there is some additional feedback provided to the submitter - it shows that someone has looked on it with at least some criticality (page hits would give you an idea as well, but doesn't mean that someone actually spent any time looking at it). Maybe instead of "dislike", it's just a "needs work" vote which has zero score effect instead of negative. I don't think having dislike negative scores will change the over rankings much anyway - something with no likes is in the same boat as a page with a few dislikes.

So, maybe "like/needs work" as the two options.

I disagree. I think that there needs to be some mechanism for a negative vote to be registered. Pages that are just bad need to just get bad votes. If the 'initial construction period' passes and the page is still bad, it needs to get dinged.

Secondly, it might be good if the "needs work" votes sort of dissolved over time, maybe 6-12 months. That way, no one has to go begging for old votes to be removed/changed. It wouldn't change the rankings much since pages still need lots of "like" votes to score high. I imagine some folks might like a "please revisit" button that can be used to ping "needs work" voters to reconsider if work has been done to improve it. I could see this being abused, but it might be worth a try.

In my model with the 'abstain' choice, those that have pressed this button ought, once having done so, receive a reminder to revisit the page after a certain amount of time has elapsed. I think we have the something similar in mind here.

Thirdly, things garner more votes when sitting in the "What's New" page, so it becomes important to some, *very* important to others. I think votes that come later after some one has stumbled upon a page or picture and taken the time to vote are more valuable as they're not just reacting to the newness factor. The voter's not just piling on when a page first gets submitted. I think there might be more weighting given to votes that come at a later time. Maybe cut it off at some point, like a year or two to keep the very oldest pages from having an advantage with this.

I think this is a really good idea. Could this be installed retroactively, i.e for votes cast in 2011 on pages created in 2007?

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 1:17 am
by Bob Burd
Montana Matt wrote:
Bob Burd wrote:So, maybe "like/needs work" as the two options. Secondly, it might be good if the "needs work" votes sort of dissolved over time, maybe 6-12 months. That way, no one has to go begging for old votes to be removed/changed.

If I understand you correctly, the "needs work" wouldn't actually affect the score, right? Is there really a need for this "needs work" button. Couldn't people just leave a note on the additions/corrections or a comment? I mean, fewer than 3% of people actually vote anything other than a 10. Seems like an unnecessary (and likely mostly unused) complication to have another option for that 3% of the time when there is already a fully functional way of giving people details (instead of a non-descriptive vote) about why you aren't voting on their page.


It was brought up that if this second option didn't reduce a score, more folks would use it, so it should become more than just 3%. It give the feedback that basically says, "I've looked at your page and it needs some work". Without it, a person might feel like nobody saw their page because it was submitted at night or whatever. Of course anyone could leave comments or an addition/corrections, but that rarely happens because it takes time beyond the average attention/care span. Let's face it, the easier and quicker the feedback mechanism, the more it will get used.

Montana Matt wrote:
Bob Burd wrote:Thirdly, things garner more votes when sitting in the "What's New" page, so it becomes important to some, *very* important to others. I think votes that come later after some one has stumbled upon a page or picture and taken the time to vote are more valuable as they're not just reacting to the newness factor. The voter's not just piling on when a page first gets submitted. I think there might be more weighting given to votes that come at a later time. Maybe cut it off at some point, like a year or two to keep the very oldest pages from having an advantage with this.

That's an interesting idea. The time of the vote is already saved in the database, as is the time of creation of a page. Something could be included in the weighting to include the time the vote was cast relative to the age of the page. So, for example, perhaps votes that occur between 0 and 30 days after creation get weighted by 0.75, votes between 30 and 500 days get weighted 1.0 and votes after 500 days get weighted 0.75? Though I prefer the idea of weighting based on page hits over this.


This wasn't suggested as a way to unweight popular pages. That was the next suggestion. This was a way to make all those initial votes garnered because they're on the What's New page a little less valuable than those that come later, usually with more thought and heart behind them. Something like 0.5 weight for first week, 0.75 up to 3 months, 1.0 beyond that, was closer to what I had in mind.

Montana Matt wrote:
Bob Burd wrote:Ok, lastly (for now), perhaps add an inverse weighting based on the number of summit logs (or perhaps page hits), with the intent to keep pages like Rainier and Everest from popping to the top based solely on popularity.

That's another interesting idea.

I can certainly play around with these things and see what I can come up with. My biggest question at this point is how to actually determine and display the score based on the number of "likes." So that problem is two fold:
1) How to weight the votes? Power will definitely play a role, as it always has, but should we consider other things as well, such as Bob is proposing? I really like the idea of using the number of page hits to weigh down the vote for pages with a lot of hits.
2) How to calculate the actual page score? Would more people prefer to see a simple cumulative number, like power, or would more people prefer a percentage. If it were a percentage, it would probably be something like the
Code: Select all
($sum_votes/$sum_max_votes)*100

Where $sum_votes is the weighted sum of the votes for the current page and $sum_max_votes is the sum of the votes for page with the highest sum.


I think having an open-ended score will make more sense to more people. Let people see that their vote adds something substantive, even if only a few points rather than a few hundredths of a percent.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 1:52 am
by Josh Lewis
Bob Burd wrote:That was the next suggestion. This was a way to make all those initial votes garnered because they're on the What's New page a little less valuable than those that come later, usually with more thought and heart behind them. Something like 0.5 weight for first week, 0.75 up to 3 months, 1.0 beyond that, was closer to what I had in mind.


I definitely agree with this statement. :)

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 3:31 am
by mrchad9
Matt- please give me two days to get a algorithm to you to consider. I've been thinking about this for some time and think I could propose something on how the page scores would be calculated that would clearly allow folks to use more of the voting scale comfortably.

I question the reasoning behind making scores go down much because votes are old. Voting on a page a year ago doesn't make it any worse today. Also this penalizes most of the types of peaks that are submitted today. Big name peaks were submitted long ago, and continue to get votes. But nowadays a new submission might get 20 votes the first two weeks and one or two over the next two years. This is also why I think using a vote to hit ratio could become problematic. Both using a ratio and time make more sense if users were forced to vote to some degree, but that has the potential to hinder ease of browsing.

Please give me till Monday or Tuesday night. I'd really like to see the full scale work and have an idea for the algorithm I just need to get the specific formula.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 4:17 am
by mrchad9
Montana Matt wrote:
Bob Burd wrote:Perhaps what people are looking for is an open-ended scoring system, rather than the current 0-100%. Score would then be something like: sum(weight1*vote1, weight2*vote2,etc)

I think you may be right Bob. And we already do that calculation on the way to calculating the score (that sum of weighted votes is used in the calculation), so it wouldn't be hard to go that route. Then the score of a page would be similar to the way a user's power works now, with no finite end.
Bob Burd wrote:The goal is still the same - trying to sort good from bad. Currently, the page with the most 10 votes pops up on top. The above scoring would pretty much do the same, I think.

Yes, at this point it would be pretty much identical. As you say, the page with the most 10 votes is on top, followed by the page with the next most 10 votes, etc. So maybe the "score" of the page could simply be the sum of the weighted votes...that would be easy enough to implement and wouldn't require as significant of a change to the database, code and HTML.
Bob Burd wrote:I'm not sure what the advantage is of keeping scores in the 0-100% range. Does it do a better job of sorting for the very best somehow?

No. I'm not sure why we decided on the % at the end, but we decided to map it to 0-100. The score calculation yields a number between 0 and 1. I guess we assumed that most people would rather see a whole number than a decimal, so we multiply the score by 100 to get something between 0 and 100.
Bob Burd wrote:btw, I think weighting should be a significant factor. It offers some sort of quality check and keeps the creation of fake avatars for the sole purpose of voting. Having a waiting period on new members weighting may only delay this. I think some sort of participation (it could even be in the forums) should be a factor, aside from just voting.

I agree as well. I intend to keep the weighting in place as it is now.

Note however, that while an open ended scoring system accomplishes some of the goals, it lacks the ability to cast negative votes or decrease a page score. While this is infrequently used it is probably still good to incorporate. I'll have a algorithm to propose soon.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 4:22 am
by mrchad9
Matt when considering hits, and penalizing pages that have hits but not votes, note that many of my pages for example may get 1000s of hits per years, but only ONE vote over that period. If anything I think more hits should add to a score, not take away. I think better to do nothing as there may be in intended consequences once put into place.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 4:46 am
by mrchad9
Ok... But is there anything wrong with high profile pages that get the most Google hits having the highest score? I think we are trying to solve a problem with this aspect of a potential change that doesn't exist. I think it is ok that they get more votes. Yes... It would be better if it were a completely level playing field, but in general this is a small problem. Should we also adjust page scores down if they had the opportunity to be featured on the home page (thus getting many more votes than they otherwise would?). I think there is a big can of worms here and the fact that Mount Shasta, Rainier, and Whitney have tons of votes at the moment likely is't hurting anything!

I suppose I would just suggest being extra careful that there are no unintended side effects when solving a problem like this since the need for it appears to be small. There were significant unexpected effects when the existing 10 point voting system was established.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 5:07 am
by Sarah Simon
Ooof, I just stumbled upon this thread. Our community's scenario regarding the scoring system and potential changes reminds me of the scenario I've witnessed over and over again in my business life where companies spend massive quantities of money, effort, personnel hours, blood, sweat and tears only to trade an old set of problems for a shiny new set of problems.

I don't pretend to had the answers, but let's not allow our urgency to "do something" cause us fresh headaches.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 5:25 am
by Bob Burd
Kids do the same thing all the time Sarah. They watch an anthill for a while, noticing the patterns and methods by which ants go about their business, marveling at micro-engineering and their doggedly persistent cooperation. Getting a little bored, they step all over the ants or shove a stick in the hole, then see how they react. :D

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 5:54 am
by mrchad9
I agree Sarah. My initial proposal will include as few changes as possible, keeping the percentage score and only attempting to change things so that 7-10 can always increase scores to varying degrees (and perhaps lowering overall starting points for page with only one or two votes). It seems to be if you have only one 10 (or whatever) that 75% is a more appropriate score. This will help get us away from every page score being between 86 and 91%.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 10:16 am
by mrchad9
I've got the formula Matt!

This wasn't easy, and unfortunately isn't the sort of thing that I can sleep over, but figured it out. And it works. I do hope you consider it.

There are variables that can be tweaked which we could discuss, such as the starting score and how quickly page scores approach 0% or 100%. These are very easily adjusted. Currently the starting score is 70%, and it takes 28 10/10 votes to reach 90%, 68 to reach 98%. This is a bit fast, unless folks really start using the 6-9 range. If most votes remain 10 I would adjust the formula.

Implementing this completely resolves people's voting issues and minimizes the risk of any unintended side effects (which is why I kept using the result of a percentage score for each page).

The formula... it looks complicated but is should be easy to implement and calculate.

Page score = 70 + 30 * [1-1/e^((x-5.5)*y/5.555555)]

e = 2.718281828459
x = simple weighted average page score between 1 and 10 based on voter weighting (if all voters give a 10 then this average is a 10)
y = total number of votes for a page


It works very well. Also note that while the score allows some members to be weighted more than others (seperate discussion) it does not affect the page score itself (i.e. if members with low weightings vote on one page and high ranking on another, but both pages have the same number of votes, then they get the same score. Member weighting only affects how your vote is considered versus another member voting on the same page).

Here are some examples...

One 10/10 = 71.18%
One 6/10 = 70.13%
Ten 10/10 = 79.89%
Twenty 10/10 = 85.62%
Thirty 10/10 = 90.96%
Thirty 10/10 and one 6/10 = 91.00%
Thirty 10/10 and one 8/10 = 91.16%
Thirtyone 10/10 = 91.32%
One hundred 10/10 = 99.451%
One hundred 10/10 and one 6/10 = 99.453%
Ten 8/10 = 75.98%
Thirty 8/10 = 84.60% (same as 17 10/10 votes)
Thirty 6/10 = 73.74%
Thirty 6/10 and one 10/10 = 74.77%
Ten 4/10 = 65.72%
Twenty 2/10 = 44.11%

The key is basically that multiple votes of 6 or 7 etc are equal to fewer votes of 9 or 10 (as in the above example 30 votes of 8 = 17 votes of a 10). The score will gradually but continuously approach 100% as long as votes are anything equal to or above 6.

I think this is much closer to realizing the desired A, B, C, D, F page scores people are inclined to think of, and under ALL circumstances a 6, 7, 8, 9, or 10 will increase a pages score (and a 1-5 will lower it).

What do you think? If you want to discuss or play around with the variables you have my contact information...

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 5:36 pm
by mrchad9
Just to explain the above formula slightly but at a high level...

It basically starts with pages at a 70% score and based on the number of votes most pages get makes an effort to have a nice distribution from 70-95% or so for pages that get positive votes.

The score asymptotically approaches 100%. Each 10 vote moves the score about 4% closer to 100. An 8 counts about half what a 10 does, a 9 about 3/4ths, a 7 about 1/3rd, and a 6 about 10% of a 10 (but still positive).

Here is a chart that shows how page scores increase when recieving a series of up to 40 votes that are all 10s, 9s, etc...

chart.jpg
chart
chart.jpg (74.1 KiB) Viewed 2132 times

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 6:16 pm
by Bob Burd
Nice contribution Chad. Overall, I think it may be a better way to do the 0-100% scoring than the current algorithm. One characteristic that could use some help - low scores should drop off more quickly. The reason for this is that if a page is lame, it gets only a few crappy votes before it is ignored and gets no more votes. With the above scheme, it will hover around 70%. It is unlikely that any page will garner 20+ poor votes. This makes the worst page hard to distinguish from a good page with only a few votes.

I still favor an open-ended scoring. Other things being equal (in regards to popular peaks vs obscure ones, for example), a page with 20 good votes really does seem like it's twice as good as one that garners 10. It might be that I just like seeing a bigger spread in the scoring.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 6:59 pm
by mrchad9
Thanks for the comment Bob. If folks determine the mechanism is acceptable, I could modify the formula to incorporate any desired changes (probably with a smaller group of decision makers such as yourself and a couple others rather than trying to do everything by committee). I am happy to make changes if this is going to be used.

An easy modification would be for initial scores to be much lower so scores could be distributed over a large spread as you indicate. The formula I proposed above would lead to most pages being between 75-90 (still much better than the current 87-91). It would be simple to make page scores start at 50% or some other figure, and climb more rapidly initially. This would automatically result in low scores decreasing a page value more rapidly too.

Also, if it were decided we want to do so, I could determine how to adjust the formula so that page scores decrease dramatically faster when getting low votes as opposed to how high votes are treated. This might involve adding a second arguement to the formula (not necessarily, but I think that is the easiest way to design it), so I would want to discuss it with Matt first so that I don't spend time to incorporate any components that might be slow to calculate and I know what the limitations are in advance. I mainly wanted to go ahead and prove the concept here, and if it is decided to do this then we can define and include additional requirements and make this as sophisticated as desired.

Re: Changes to Voting System

PostPosted: Mon Jan 14, 2013 7:04 pm
by Scott
It is unlikely that any page will garner 20+ poor votes.


Also, if it were decided we want to do so, I could determine how to adjust the formula so that page scores decrease dramatically faster when getting low votes as opposed to how high votes are treated.


Yes. It must be remembered that even a poor page (even if nearly blank) is still going to gather a lot of positive votes (and a new voting system won't change that).