Changes to Voting System

Suggestions and comments about SummitPost's features, policies, and procedures. Post bugs here.
User Avatar
Scott
Forum Moderator
 
Posts: 8548
Joined: Thu Aug 21, 2003 1:03 pm
Thanked: 1211 times in 649 posts

Re: Changes to Voting System

by Scott » Wed Jan 16, 2013 7:42 pm

Chad and Matt,

This is just a suggestion and only my opinion, but in a way, rather than have the page value vs. page score (at least for mountain pages/routes) be more linear, or perhaps even using an exponent less than one, so value would level off at some point.

Having it exponential at the beginning can be good because poor pages still receive positive votes, but usually not as many.

The only reason I propose having the value level off at the end rather than to be exponential is because it creates a bias for high traffic pages and against new members.

Just about any mountain page (there probably is the rare exception) that is going to be high traffic has already been added to SP, probably years ago. As time passes, more and more lesser known mountains will be added (since the available well known mountains to add will shrink as time goes on).

Personally, from building my own pages I think that it is much harder to create a really good page on a lesser known mountain than it is with a well-known one.

I recently inherited a few pages for a well know mountain from a member whom deleted his/her content. Rebuilding it was easy because I could rely on not only not only on information gathered from my own climb, but check many online sources/guidebooks, etc. for more information, to refresh memory, etc. The mountain had several well-known routes as well and it was easy to list them.


On the other hand, when you create a page on an obscure mountain, it’s a lot more work. You have only your own personal experience to rely on. If there is any other information out there, you will really have to dig for it. You have to make sure your information is especially accurate because there are no other sources to check your memory against.

Random examples of two equally difficult and spectacular mountains in their own way:

http://www.summitpost.org/capitol-peak/150528

http://www.summitpost.org/morne-trois-pitons/770386

The first page was easy to write for the reasons listed above. The second page, although it appears to have less information was actually harder and more work to create.

On the front page, good pages will typically get more votes than bad pages, but once off the front page, the number of votes is fairly non-relevant and high traffic pages on well-known mountains will always receive the most votes (for obvious reasons). A page with 10 votes doesn’t always mean that it’s higher quality than a page with 50 votes; it just means that more people were searching for that mountain.

Of course this isn’t a complaint, only a suggestion and not a pressing issue.

The following user would like to thank Scott for this post
rgg

User Avatar
Bob Sihler
Forum Moderator
 
Posts: 8486
Joined: Mon Aug 23, 2004 3:20 pm
Thanked: 2763 times in 1527 posts

Re: Changes to Voting System

by Bob Sihler » Wed Jan 16, 2013 8:58 pm

Montana Matt wrote:Good suggestion Scott.
Scott wrote:On the front page, good pages will typically get more votes than bad pages, but once off the front page, the number of votes is fairly non-relevant and high traffic pages on well-known mountains will always receive the most votes (for obvious reasons). A page with 10 votes doesn’t always mean that it’s higher quality than a page with 50 votes; it just means that more people were searching for that mountain.

Then maybe a good solution would be to feature more obscure pages that get less traffic but are very well done? While at the same time, trying to avoid featuring pages that are already getting a lot of traffic/votes?


That is exactly one of the things I am going to be doing.
"Alcohol is like love. The first kiss is magic, the second is intimate, the third is routine. After that you take the girl's clothes off."

--Terry Lennox, The Long Goodbye (Raymond Chandler)

User Avatar
Bubba Suess

 
Posts: 726
Joined: Tue Oct 04, 2005 9:15 pm
Thanked: 183 times in 105 posts

Re: Changes to Voting System

by Bubba Suess » Wed Jan 16, 2013 8:59 pm

Just out of curiosity, when the new voting regime is implemented, how will all the previous votes count? Will they still count as 10/10, and all the pages just have really high scores?

User Avatar
Buz Groshong

 
Posts: 2845
Joined: Tue Sep 14, 2004 10:58 pm
Thanked: 687 times in 484 posts

Re: Changes to Voting System

by Buz Groshong » Wed Jan 16, 2013 9:00 pm

Sarah Simon wrote:
Bubba Suess wrote:This is totally nit-picking, but ought canyons at least be equivalent to Mountains and Areas? Keep the physical features at the same level. Pages like this, which has lots of climbing beta should be worth as much as a mountain.


Bubba, while I agree that the canyon pages add value to this site, I also believe that canyons are ancillary to the purpose of this site - SummitPost. If we keep physical features at the same contribution level, does that mean we start adding Rivers and thus develop a PaddlePost element to the site? It's important that we maintain focus.


Focus on what? Most of our members are hikers rather than climbers, and canyons are just as important to many of us as mountains are. We should bear in mind that we have various things that are called "mountains" that really aren't. Two that readily come to mind are http://www.summitpost.org/blackrock-big-meadows/230710 and http://www.summitpost.org/buck-ridge/405273. Then there's the question of "routes;" we have various kinds (alpine, rock climbing, and hiking). Perhaps we should first define what SummitPost is about - maybe it should be the first item in FAQ or be stated on the "About" page.

User Avatar
mrchad9

 
Posts: 4545
Joined: Thu Jul 02, 2009 12:01 am
Thanked: 1338 times in 911 posts

Re: Changes to Voting System

by mrchad9 » Wed Jan 16, 2013 9:05 pm

Scott wrote:Chad and Matt,

This is just a suggestion and only my opinion, but in a way, rather than have the page value vs. page score (at least for mountain pages/routes) be more linear, or perhaps even using an exponent less than one, so value would level off at some point.

Good suggestion... but this is the case Scott!

OK... well not page value vs page score... but page value vs number of votes. Using an exponent less than one happens to make page value vs page score pretty linear. With the changes to the page values, I was trying to reward highly scoring pages, which I think most agree deserve more credit. Still all good pages will receive a consequential number of points. While the system may not be perfection, I think this design will drive the correct behaviors (encourage quality).

Note that page scores level off. True that page value vs score does not level off (I have at least capped them and removed the exponential component of them, which based on your post you would have a tremendous issue with the system we have been living with).

But since page scores level off (and page values are driven off this) as pages score more votes the impact on points decreases continuously.

Note the graph below. As you can see as pages reach 30-40 votes (which is what seperates really good pages with really popular mountains) the number of points decreases significantly as more votes are aquired. By the time you have 49 votes you can only get 1 more point (14% more credit) even if your page got 1 million more votes after that. After getting 30 votes you only have the potential to get 25% more credit with 1 million more votes.
Attachments
pointsvsvotes.jpg
Points as a function of vote quantity
pointsvsvotes.jpg (49.55 KiB) Viewed 1345 times

The following user would like to thank mrchad9 for this post
chugach mtn boy

User Avatar
mrchad9

 
Posts: 4545
Joined: Thu Jul 02, 2009 12:01 am
Thanked: 1338 times in 911 posts

Re: Changes to Voting System

by mrchad9 » Wed Jan 16, 2013 9:09 pm

Bubba Suess wrote:Just out of curiosity, when the new voting regime is implemented, how will all the previous votes count? Will they still count as 10/10, and all the pages just have really high scores?

I would think they remain 10s. Note that scores will decrease a lot though, as they would be part of the new process that scores pages in the 70-90% range for the most part. But they would have some advantage over newer pages, since new pages will get votes in the 6-10 range.

I think the cost is small. Good behaviors will be rewarded in the future. The resolution to this, however, would be for Matt to discount the weighting of votes prior to the transition relative to any new votes a page might get. Easy to do with the future mechanism if it is decided to do so.

User Avatar
Bubba Suess

 
Posts: 726
Joined: Tue Oct 04, 2005 9:15 pm
Thanked: 183 times in 105 posts

Re: Changes to Voting System

by Bubba Suess » Wed Jan 16, 2013 9:17 pm

mrchad9 wrote:I would think they remain 10s. Note that scores will decrease a lot though, as they would be part of the new process that scores pages in the 70-90% range for the most part. But they would have some advantage over newer pages, since new pages will get votes in the 6-10 range.

That is fine. I was just curious how that was going to transition.

User Avatar
MoapaPk

 
Posts: 7780
Joined: Fri May 13, 2005 7:42 pm
Thanked: 787 times in 519 posts

Re: Changes to Voting System

by MoapaPk » Wed Jan 16, 2013 9:23 pm

I would point out that sometimes a simple topographic map (which is a picture), or lowly picture with a route drawn on, is more valuable than a long mountain "page."

http://www.summitpost.org/thimble-peak/344399

vs

http://www.summitpost.org/direct-way-up ... 2/c-344396
Last edited by MoapaPk on Wed Jan 16, 2013 9:28 pm, edited 2 times in total.

The following user would like to thank MoapaPk for this post
MarkDidier

User Avatar
mrchad9

 
Posts: 4545
Joined: Thu Jul 02, 2009 12:01 am
Thanked: 1338 times in 911 posts

Re: Changes to Voting System

by mrchad9 » Wed Jan 16, 2013 9:26 pm

MoapaPk wrote:or lowly picture with a route drawn on, is 10x more valuable than a long bullshit mountain "page."

You mean like this?

http://www.summitpost.org/c-closer-view ... ute/271543

Image

User Avatar
Scott
Forum Moderator
 
Posts: 8548
Joined: Thu Aug 21, 2003 1:03 pm
Thanked: 1211 times in 649 posts

Re: Changes to Voting System

by Scott » Wed Jan 16, 2013 10:23 pm

Then maybe a good solution would be to feature more obscure pages that get less traffic but are very well done? While at the same time, trying to avoid featuring pages that are already getting a lot of traffic/votes?


Yes.

but this is the case Scott!

Using an exponent less than one happens to make page value vs page score pretty linear.


But since page scores level off (and page values are driven off this) as pages score more votes the impact on points decreases continuously.

By the time you have 49 votes you can only get 1 more point (14% more credit) even if your page got 1 million more votes after that.


Thanks for the graph/explanation. Personally, after the page has been on the front page; I don't think the number of positive votes has that much relevance (though I guess many might disagree). Maybe for the first dozen votes or so, but not much after that. This can even be more the case for things like adopted pages, obscure mountains vs. popular, etc.

As a random example, what would you say the actually quality difference between the two pages below would be?

http://www.summitpost.org/tabeguache-peak/150374

http://www.summitpost.org/frisco-peak/287560

Of course that is an extreme example, and perhaps I am somewhat biased because I typically find the pages on obscure mountains/routes most interesting and useful (especially if SP is the only source for the information). (On a different note, the number of recent negative votes usually is very relevant because it may point out abandoned pages, uncorrected errors, etc.)

I have at least capped them and removed the exponential component of them, which based on your post you would have a tremendous issue with the system we have been living with


Yes, yours is better.

In the end, this isn't much of an issue.

The following user would like to thank Scott for this post
mrchad9

User Avatar
mrchad9

 
Posts: 4545
Joined: Thu Jul 02, 2009 12:01 am
Thanked: 1338 times in 911 posts

Re: Changes to Voting System

by mrchad9 » Wed Jan 16, 2013 10:26 pm

Scott wrote:As a random example, what would you say the actually quality difference between the two pages below would be?

http://www.summitpost.org/tabeguache-peak/150374

http://www.summitpost.org/frisco-peak/287560

We need a good mechanism to help identify and highlight pages that have been fixed up and improved. Will have to think on that.

The following user would like to thank mrchad9 for this post
Scott

User Avatar
Bruno

 
Posts: 278
Joined: Tue Oct 02, 2007 9:16 am
Thanked: 112 times in 76 posts

Re: Changes to Voting System

by Bruno » Wed Jan 16, 2013 11:58 pm

Thanks a lot Matt, Chard, Bob, Scott and everybody trying to fix the current voting system.

I like the new proposed algorithms, especially the change in the weighting of the different objects (less importance given to the pictures).

Thinking about the psychology of the average voter (myself included), I don't think the proposed calculation with every vote above 5 increasing the page score would help establishing a better hierarchy in page quality. To the opposite, popular pages with high number of votes will invariably be on top, as almost nobody will give a vote lower than 5.

I'm ready to bet that we will continue having >96% of votes being 10/10, so the new algorithm might not change anything… And even if a few more 6/10, 7/10, 8/10 or 9/10 are given, this won't affect the "ranking" of existing pages, which will continue to be mainly based on the existing 10/10 votes.

Actually, most of the few current votes in the 6/10-9/10 range were meant to reduce the page score and politely invite the owner to update/revise his/her page. The new algorithm might simply remove the effect of these votes, pushing the ranking of such pages that need to be fixed even higher.

It is actually very difficult to introduce a new voting system that is not an indicator of popularity (instead of quality), especially if we are keeping the millions of existing 10/10 votes. Whatever new algorithm you introduce, it won't affect much the existing pages. This was the reason why I was favouring a dual system ("Good / To be improved") hoping that more members would dare to use the "To be improved" option. But maybe the system can't be fixed, or doesn't even need to be fixed, as you don't really look for mountains and routes based on page score...

Anyway, thanks again for the great work and proposals.

User Avatar
Josh Lewis

 
Posts: 3412
Joined: Thu Jan 06, 2011 11:12 pm
Thanked: 1108 times in 677 posts

Re: Changes to Voting System

by Josh Lewis » Thu Jan 17, 2013 12:34 am

@ Bruno : When the new voting system is in, lets go to town voting honestly. Then when our inbox's get flooded with "Why did you vote a 8/10?" we can explain about the new voting system. I think when SP V3 is out, it will be obvious that things changed and there will probably be an announcement of some sort.

User Avatar
mrchad9

 
Posts: 4545
Joined: Thu Jul 02, 2009 12:01 am
Thanked: 1338 times in 911 posts

Re: Changes to Voting System

by mrchad9 » Thu Jan 17, 2013 1:02 am

Thanks for the kind words on the weighting Bruno. You always have thoughtful opinions.

The method of scoring a page is a difficult problem to solve, and I don't think the proposed algorithm is a perfect solution, but I do think it is the best solution possible. There probably isn't a perfect solution, and it is at least a nice step forward. People will still get to use votes to encourage better pages... votes can be 1-5 too, and I think many folks will strive to do quality work so they get more 9-10s if they see they are getting 6-7s. At least I hope so.

I bet you are correct that tons of 10s still get doled out... but I bet most of the thoughtless 10s start going to photos, which frankly I don't really care about. Voting on pics IS a photo popularity contest, and there is really no point to telling someone with an 8 that a photo is ok but not so perfect. But I bet we do start seeing a lot more than 4% of votes being 1-9s on mountain pages... and having that feedback mechanism is really the point. Time will tell.

I appreciate your comments... Good to hear from you!

User Avatar
Bob Burd
Forum Moderator
 
Posts: 4271
Joined: Sun Jul 22, 2001 10:42 pm
Thanked: 572 times in 296 posts

Re: Changes to Voting System

by Bob Burd » Thu Jan 17, 2013 1:40 am

Bruno wrote:It is actually very difficult to introduce a new voting system that is not an indicator of popularity (instead of quality), especially if we are keeping the millions of existing 10/10 votes. Whatever new algorithm you introduce, it won't affect much the existing pages. This was the reason why I was favouring a dual system ("Good / To be improved") hoping that more members would dare to use the "To be improved" option. But maybe the system can't be fixed, or doesn't even need to be fixed, as you don't really look for mountains and routes based on page score...


One of the suggestions was to use the page hit count to inverse weight the score. In a balanced world, this would mean a page with lots of hits and lots of votes would rank similarly to a page with fewer hits and fewer votes, all else being equal. It would be interesting to see how this might work on the existing database. It might have the interesting effect of drawing obscure pages to attention where they might get more hits (and hopefully more votes) and thus become less obscure - leaving room for more obscure pages to rise to the top.

Matt - if page count were to be used, could it be page count based on unique IP addresses? It would be very easy to write a script to make many hits on a page to drop its score. Sadly, this would be inevitable if there was no mechanism in place to counter it.

PreviousNext

Return to Site Feedback

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 0 guests