Welcome to SP!  -
Areas & RangesMountains & RocksRoutesImagesArticlesTrip ReportsGearOtherPeoplePlans & PartnersWhat's NewForum

Changes to Voting System

Suggestions and comments about SummitPost's features, policies, and procedures. Post bugs here.
 

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 4:22 am

Matt when considering hits, and penalizing pages that have hits but not votes, note that many of my pages for example may get 1000s of hits per years, but only ONE vote over that period. If anything I think more hits should add to a score, not take away. I think better to do nothing as there may be in intended consequences once put into place.
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

Re: Changes to Voting System

Postby Montana Matt » Mon Jan 14, 2013 4:35 am

mrchad9 wrote:Note however, that while an open ended scoring system accomplishes some of the goals, it lacks the ability to cast negative votes or decrease a page score.

That's somewhat true. But there are quite a few ways that page score could drop with the system that I'm considering most right now.
1) People can remove their votes (if a page lacks maintenance and the beta is extremely outdated, for example) and that will lower the page score.
2) Though rare if contributing regularly, people's power can decrease, which will also affect all of the weight of their votes in the system, thus potentially decreasing the page score of anything they've "liked".
3) Pages that get a lot of hits but not many "likes" will decrease in score over time.
mrchad9 wrote:Matt when considering hits, and penalizing pages that have hits but not votes, note that many of my pages for example may get 1000s of hits per years, but only ONE vote over that period.

The "hits" you see displayed on the page are including search engine traffic an non-SP users. I'm talking about only counting hits for people logged into the system with the ability to vote. I think over time it will average out so that all pages are viewed proportionally equally by SP-users (non-SE traffic and non-anonymous users).
mrchad9 wrote:If anything I think more hits should add to a score, not take away. I think better to do nothing as there may be in intended consequences once put into place.

If all hits (logged in and anonymous) are factored into the score, pages that rank best in Google are going to have the highest score. We're trying to devise a way to give all pages an equal chance to score well regardless of how popular a peak or a route (or whatever type of page) is. I'm making the assumption that in most cases, popular mountain pages aren't visited by SP members (when they're logged in) any more frequently than other pages that are less popular. Or rather, that the percentage of votes cast vs. page visits when people are logged into the system, regardless of page popularity, is fairly consistent. I would guess that to be an accurate assumption as people who think a page is good vote 10 and if they don't like it, they don't vote.
User Avatar
Montana Matt
Site Admin
 
Posts: 1125
Joined: Sun Apr 11, 2004 10:06 pm
Location: Ashland, Oregon, United States
Thanked: 340 times in 184 posts

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 4:46 am

Ok... But is there anything wrong with high profile pages that get the most Google hits having the highest score? I think we are trying to solve a problem with this aspect of a potential change that doesn't exist. I think it is ok that they get more votes. Yes... It would be better if it were a completely level playing field, but in general this is a small problem. Should we also adjust page scores down if they had the opportunity to be featured on the home page (thus getting many more votes than they otherwise would?). I think there is a big can of worms here and the fact that Mount Shasta, Rainier, and Whitney have tons of votes at the moment likely is't hurting anything!

I suppose I would just suggest being extra careful that there are no unintended side effects when solving a problem like this since the need for it appears to be small. There were significant unexpected effects when the existing 10 point voting system was established.
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

Re: Changes to Voting System

Postby Sarah Simon » Mon Jan 14, 2013 5:07 am

Ooof, I just stumbled upon this thread. Our community's scenario regarding the scoring system and potential changes reminds me of the scenario I've witnessed over and over again in my business life where companies spend massive quantities of money, effort, personnel hours, blood, sweat and tears only to trade an old set of problems for a shiny new set of problems.

I don't pretend to had the answers, but let's not allow our urgency to "do something" cause us fresh headaches.
Go climb a mountain
User Avatar
Sarah Simon

 
Posts: 935
Joined: Tue Jan 02, 2007 12:01 am
Location: Black Forest, Colorado, United States
Thanked: 238 times in 107 posts

The following user would like to thank Sarah Simon for this post
Dow Williams, Marcsoltan, Montana Matt, rockthrowjoe, Silvia Mazzani

Re: Changes to Voting System

Postby Bob Burd » Mon Jan 14, 2013 5:25 am

Kids do the same thing all the time Sarah. They watch an anthill for a while, noticing the patterns and methods by which ants go about their business, marveling at micro-engineering and their doggedly persistent cooperation. Getting a little bored, they step all over the ants or shove a stick in the hole, then see how they react. :D
User Avatar
Bob Burd
Forum Moderator
 
Posts: 4097
Joined: Sun Jul 22, 2001 10:42 pm
Location: San Jose, California, United States
Thanked: 458 times in 237 posts

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 5:54 am

I agree Sarah. My initial proposal will include as few changes as possible, keeping the percentage score and only attempting to change things so that 7-10 can always increase scores to varying degrees (and perhaps lowering overall starting points for page with only one or two votes). It seems to be if you have only one 10 (or whatever) that 75% is a more appropriate score. This will help get us away from every page score being between 86 and 91%.
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

The following user would like to thank mrchad9 for this post
Sarah Simon

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 10:16 am

I've got the formula Matt!

This wasn't easy, and unfortunately isn't the sort of thing that I can sleep over, but figured it out. And it works. I do hope you consider it.

There are variables that can be tweaked which we could discuss, such as the starting score and how quickly page scores approach 0% or 100%. These are very easily adjusted. Currently the starting score is 70%, and it takes 28 10/10 votes to reach 90%, 68 to reach 98%. This is a bit fast, unless folks really start using the 6-9 range. If most votes remain 10 I would adjust the formula.

Implementing this completely resolves people's voting issues and minimizes the risk of any unintended side effects (which is why I kept using the result of a percentage score for each page).

The formula... it looks complicated but is should be easy to implement and calculate.

Page score = 70 + 30 * [1-1/e^((x-5.5)*y/5.555555)]

e = 2.718281828459
x = simple weighted average page score between 1 and 10 based on voter weighting (if all voters give a 10 then this average is a 10)
y = total number of votes for a page


It works very well. Also note that while the score allows some members to be weighted more than others (seperate discussion) it does not affect the page score itself (i.e. if members with low weightings vote on one page and high ranking on another, but both pages have the same number of votes, then they get the same score. Member weighting only affects how your vote is considered versus another member voting on the same page).

Here are some examples...

One 10/10 = 71.18%
One 6/10 = 70.13%
Ten 10/10 = 79.89%
Twenty 10/10 = 85.62%
Thirty 10/10 = 90.96%
Thirty 10/10 and one 6/10 = 91.00%
Thirty 10/10 and one 8/10 = 91.16%
Thirtyone 10/10 = 91.32%
One hundred 10/10 = 99.451%
One hundred 10/10 and one 6/10 = 99.453%
Ten 8/10 = 75.98%
Thirty 8/10 = 84.60% (same as 17 10/10 votes)
Thirty 6/10 = 73.74%
Thirty 6/10 and one 10/10 = 74.77%
Ten 4/10 = 65.72%
Twenty 2/10 = 44.11%

The key is basically that multiple votes of 6 or 7 etc are equal to fewer votes of 9 or 10 (as in the above example 30 votes of 8 = 17 votes of a 10). The score will gradually but continuously approach 100% as long as votes are anything equal to or above 6.

I think this is much closer to realizing the desired A, B, C, D, F page scores people are inclined to think of, and under ALL circumstances a 6, 7, 8, 9, or 10 will increase a pages score (and a 1-5 will lower it).

What do you think? If you want to discuss or play around with the variables you have my contact information...
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

The following user would like to thank mrchad9 for this post
lcarreau, rockthrowjoe, Sarah Simon, yatsek

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 5:36 pm

Just to explain the above formula slightly but at a high level...

It basically starts with pages at a 70% score and based on the number of votes most pages get makes an effort to have a nice distribution from 70-95% or so for pages that get positive votes.

The score asymptotically approaches 100%. Each 10 vote moves the score about 4% closer to 100. An 8 counts about half what a 10 does, a 9 about 3/4ths, a 7 about 1/3rd, and a 6 about 10% of a 10 (but still positive).

Here is a chart that shows how page scores increase when recieving a series of up to 40 votes that are all 10s, 9s, etc...

chart.jpg
chart
chart.jpg (74.1 KiB) Viewed 170 times
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

Re: Changes to Voting System

Postby Bob Burd » Mon Jan 14, 2013 6:16 pm

Nice contribution Chad. Overall, I think it may be a better way to do the 0-100% scoring than the current algorithm. One characteristic that could use some help - low scores should drop off more quickly. The reason for this is that if a page is lame, it gets only a few crappy votes before it is ignored and gets no more votes. With the above scheme, it will hover around 70%. It is unlikely that any page will garner 20+ poor votes. This makes the worst page hard to distinguish from a good page with only a few votes.

I still favor an open-ended scoring. Other things being equal (in regards to popular peaks vs obscure ones, for example), a page with 20 good votes really does seem like it's twice as good as one that garners 10. It might be that I just like seeing a bigger spread in the scoring.
User Avatar
Bob Burd
Forum Moderator
 
Posts: 4097
Joined: Sun Jul 22, 2001 10:42 pm
Location: San Jose, California, United States
Thanked: 458 times in 237 posts

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 6:59 pm

Thanks for the comment Bob. If folks determine the mechanism is acceptable, I could modify the formula to incorporate any desired changes (probably with a smaller group of decision makers such as yourself and a couple others rather than trying to do everything by committee). I am happy to make changes if this is going to be used.

An easy modification would be for initial scores to be much lower so scores could be distributed over a large spread as you indicate. The formula I proposed above would lead to most pages being between 75-90 (still much better than the current 87-91). It would be simple to make page scores start at 50% or some other figure, and climb more rapidly initially. This would automatically result in low scores decreasing a page value more rapidly too.

Also, if it were decided we want to do so, I could determine how to adjust the formula so that page scores decrease dramatically faster when getting low votes as opposed to how high votes are treated. This might involve adding a second arguement to the formula (not necessarily, but I think that is the easiest way to design it), so I would want to discuss it with Matt first so that I don't spend time to incorporate any components that might be slow to calculate and I know what the limitations are in advance. I mainly wanted to go ahead and prove the concept here, and if it is decided to do this then we can define and include additional requirements and make this as sophisticated as desired.
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

Re: Changes to Voting System

Postby Scott » Mon Jan 14, 2013 7:04 pm

It is unlikely that any page will garner 20+ poor votes.


Also, if it were decided we want to do so, I could determine how to adjust the formula so that page scores decrease dramatically faster when getting low votes as opposed to how high votes are treated.


Yes. It must be remembered that even a poor page (even if nearly blank) is still going to gather a lot of positive votes (and a new voting system won't change that).
User Avatar
Scott

 
Posts: 7438
Joined: Thu Aug 21, 2003 1:03 pm
Location: Craig, Colorado, United States
Thanked: 537 times in 302 posts

Re: Changes to Voting System

Postby mrchad9 » Mon Jan 14, 2013 7:16 pm

If Matt says we can use two arguements (use one formula for 6-10 and another for 1-5 and add the results together) then this is very easy to do to handle low votes. Otherwise I need to think on it a while. Could probably come up with something that looks cleaner and hopefully accomplishes the same result (and perhaps better in the long run).
User Avatar
mrchad9

 
Posts: 4162
Joined: Thu Jul 02, 2009 12:01 am
Location: San Ramon, California, United States
Thanked: 1196 times in 809 posts

Re: Changes to Voting System

Postby Montana Matt » Mon Jan 14, 2013 7:24 pm

mrchad9 wrote:The formula... it looks complicated but is should be easy to implement and calculate.

Page score = 70 + 30 * [1-1/e^((x-5.5)*y/5.555555)]

e = 2.718281828459
x = simple weighted average page score between 1 and 10 based on voter weighting (if all voters give a 10 then this average is a 10)
y = total number of votes for a page

Nice work Chad. I took the formula and have been running it on real data from the SP database. The results I'm getting are different from the ones you posted above.

There are almost no votes other than 10s in the system, so I really only tested with the 10s. This is what I'm seeing:
One 10 vote = 86.654259094317
Two 10 votes = 94.063039989279
Three 10 votes = 97.358895664317
Four 10 votes = 98.825083527703
Five 10 votes = 99.477328972497

Here's my code. Check it for errors:
Code: Select all
    foreach( $vote_data as $vote_d )
    {
        // Get the user's weight
        $weight = get_vote_weight($vote_d['power'], $max_power);

        $weighted_num_votes += $weight;
        $weighted_vote_sum += $vote_d['vote'] * $weight;
        $tot_votes++;
    }

    $nat = 2.718281828459;

    if( $tot_votes > 0 )
    {
        $weighted_average = $weighted_vote_sum/$weighted_num_votes;
        $exp = ($weighted_average-5.5)*$tot_votes/5.555555;
        $score = 70 + 30 * (1 - (1/(pow($nat, $exp))));
        echo "$obj_id - $obj_name - score -- > $score\n";
    }


Since everyone votes 10/10 (at least right now), for clarification purposes, the equation can basically be simplified to:
Page score = 70 + 30 * [1-1/e^(0.81*y)]
y = total number of votes for a page


Can you check your calculations again Chad? Also, how did you go about determining the two constants (5.5 and 5.555555) in that equation? If the 5.555555 is changed to a 55.55555 (which would drop that exponent from 0.81 to 0.081), I'm seeing numbers more along the lines of what you posted.

I do really like this and I think it could work well.
Bob Burd wrote:With the above scheme, it will hover around 70%. It is unlikely that any page will garner 20+ poor votes. This makes the worst page hard to distinguish from a good page with only a few votes.

This is true. Most people probably won't vote, as they choose not to now, if a page is poorly done. Perhaps starting the score closer to 50 would be better? That would allow pages that get good votes to separate further, perhaps?
mrchad9 wrote:If Matt says we can use two arguements (use one formula for 6-10 and another for 1-5 and add the results together) then this is very easy to do to handle low votes. Otherwise I need to think on it a while. Could probably come up with something that looks cleaner and hopefully accomplishes the same result (and perhaps better in the long run).

It wouldn't be a problem to code for votes between 1 and 5 and votes between 6 and 10. However, it would be more computations and thus put a little more load on the server. Voting already is fairly computationally expensive. At the very least though, we could try it and see how slow it is.
User Avatar
Montana Matt
Site Admin
 
Posts: 1125
Joined: Sun Apr 11, 2004 10:06 pm
Location: Ashland, Oregon, United States
Thanked: 340 times in 184 posts

Re: Changes to Voting System

Postby Bob Burd » Mon Jan 14, 2013 8:54 pm

The 5.5 constant is what makes votes 6 and higher add to the score, 5 and below subtract from the score. The 5.55555555 give it the rate of falloff per vote, but seems arbitrary. One or two significant digits should suffice.
User Avatar
Bob Burd
Forum Moderator
 
Posts: 4097
Joined: Sun Jul 22, 2001 10:42 pm
Location: San Jose, California, United States
Thanked: 458 times in 237 posts

Re: Changes to Voting System

Postby Montana Matt » Mon Jan 14, 2013 9:06 pm

Bob Burd wrote:The 5.5 constant is what makes votes 6 and higher add to the score, 5 and below subtract from the score.

Oh right. Sure, should have seen that...
Bob Burd wrote:The 5.55555555 give it the rate of falloff per vote, but seems arbitrary.

Yeah. In the original voting system, we had 3 similar "arbitrary" constants that were determined by some nonlinear regression analysis that Josh did on SPv1 votes (I think).
User Avatar
Montana Matt
Site Admin
 
Posts: 1125
Joined: Sun Apr 11, 2004 10:06 pm
Location: Ashland, Oregon, United States
Thanked: 340 times in 184 posts

PreviousNext

Return to Site Feedback

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 0 guests

© 2006-2013 SummitPost.org. All Rights Reserved.