“it's easy to spot temporary distortions in the reviews...”
September 21, 2017 7:47 AM   Subscribe

Valve aims to combat Steam review bombing with new 'histogram' charts by Andy Chalk [PC Gamer] “Valve looked at a few possible solutions to the problem, including the elimination of review scores altogether, although that was pretty much a non-starter since they were added in response to user demand in the first place. Thought was also given to locking down reviews temporarily when "abnormal behavior," which is to say a review-bombing campaign, was detected. But that was rejected as well, because Valve doesn't want to "stop the community having a discussion about the issue they're unhappy about, even though there are probably better places to have that conversation than in Steam User Reviews." Ultimately, the decision was to do nothing, at least with regard to the review scores themselves. Instead, Valve is now providing consumers with more information about the reviews by way of a histogram that compares the ratio of a game's positive to negative reviews over its lifetime.”

• User Reviews by Alden Kroll [Steam]
“We could change the way the Review Score is calculated, focusing on much more recent data. One of the reasons a review bomb can distort a game's Review Score for a significant period of time is because the score is based on reviews over a period of 30 days for the Recent value, and all time for the Overall value. But doing this would likely result in more fluctuation and potentially less accuracy for all games, not just review bombed ones. In the end, we decided not to change the ways that players can review games, and instead focused on how potential purchasers can explore the review data. Starting today, each game page now contains a histogram of the positive to negative ratio of reviews over the entire lifetime of the game, and by clicking on any part of the histogram you're able to read a sample of the reviews from that time period. As a potential purchaser, it's easy to spot temporary distortions in the reviews, to investigate why that distortion occurred, and decide for yourself whether it's something you care about. This approach has the advantage of never preventing anyone from submitting a review, but does require slightly more effort on the part of potential purchasers.”
• Valve's Solution To Steam Review Bombing Is To Add Charts by Nathan Grayson [Kotaku]
“So basically, Valve saw a situation in which people were manipulating data and decided to add more data. It’s the most Valve solution ever. I’m not gonna beat around the bush here: I don’t think this is a great way to stop review bombs, which most recently tanked Firewatch’s review score in the wake of the Pewdiepie controversy [Previously], but have also dinged everything from indie game Titan Souls after a beef with YouTuber Totalbiscuit to Baldur’s Gate: Siege of Dragonspear after people found out the game contained a trans character. Even with these changes in place, review bombers will still be able to exert undue influence on games’ scores, which will remain a metric that dilutes conversation around games down to overly simplistic factors like “is it long” and “does it do the graphics.” Not only that, review scores affect the way games are regarded by the Steam store’s increasingly important algorithms, meaning that review bombs can damage their chances at success in that way, too. And as ever, smaller developers will remain most susceptible to the ravages of review bombs, something that will have an at least subliminal (if not overt) influence on their creative choices and actions, given that their livelihood might be at stake.”
• Valve is ‘fixing’ its review-bombing problem by making users do more work by Adi Robertson [The Verge]
“I’m not sure what’s supposed to be “easy” about this system. If a casual customer clicks on the reviews for Firewatch and sees the recent distortion, they have to dig through posts to see whether it’s a game-breaking bug or an unrelated review-bombing. When I checked last night, the most recent comment said Campo Santo “took a stand” against PewDiePie over racism, and the highest-rated said the developer was “childish and thin-skinned” with no further explanation. Valve is asking for a lot of off-platform detective work here, especially for the many people who barely know who PewDiePie is. If customers have to stop and look up answers, they may as well ignore Steam and head straight to Google for reviews. Moreover, the whole system is temporarily poisoned. Review bombers can make sure buyers only see negative feedback in the “most helpful reviews” section, and short but genuine-seeming reviews could just be subtler attempts at manipulation. [...] Valve outright admits that review sections aren’t a good place to hold certain conversations, and that better options exist. (Games already have their own Steam discussion boards, for one thing.) But it invokes one of the internet’s most beloved principles by insisting that players should be generically allowed to “voice their opinions” and “have a discussion,” even when it’s irrelevant to a specific, structured forum”
• Valve thinks charts will negate Steam review bombing by Rachel England [Engadget]
“Each game page now features a histogram of both the positive and negative reviews submitted over the lifetime of the game. Click on any part of the histogram and you'll be able to read a sample of reviews from that period. The thinking is, as a potential purchaser, you'll be able to spot temporary distortions and then investigate why that distortion happened. Take Firewatch, for example. Its reviews fell off a cliff after Sean Vanaman, from the game's developer Campo Santo, publicly denounced PewDiePie for his troubling comments. As Kroll says, you can then "decide for yourself whether it's something you care about". The new system also has the benefit of allowing players to see how a game's reviews have evolved over time, which is particularly useful for games that operate as services. As Kroll explains, "Earlier purchasers of a game are more likely to enjoy it than later purchasers. In the pool of players who are interested in a game, the ones who are more confident that they'll like the game will buy it first, so as time goes on the potential purchasers left are less and less certain that they'll like the game. So if you see a game's reviews trending up over time, it may be an even more powerful statement about the quality of work its developers are doing."”
Steam has a “review bomb” problem—but will this new feature fix it? by Sam Machkovech [ArsTechnica]
“The first problem with today's new histograms is in their looks. Quite frankly, they're hideous and hard to parse. The histograms are designed in such a way that their blue and orange bars take up an entire maximized Steam window, yet the bars are itty-bitty in terms of pixel size, so they're simultaneously hard to click. Plus, numbers and other data points are tossed onto the X and Y axes without any lines or other design cues. They look like they were designed in Microsoft Word. Should you successfully click on the bar you want, you then need to scroll your entire window down to see the relevant reviews that load (and if you don't do so quickly enough, you don't even see the reviews load or propagate). This interface still also has a confusing, separate column of "recent reviews," which will fill up with different reviews no matter which month or year of data you might be looking at. Worst of all, because Steam's store listing pages are such a cluttered mess, you have to scroll quite a ways before even finding these charts. It's a gol-danged journey: past the official description, any Early Access explainer box, the "buy now" buttons, the DLC options, the "recent news" boxes, an "About This Game" text box, a "system requirements" box, a "more like this" ad for other games, and the "what curators say" box. And after all of that, Steam thinks we want to pick and prod at a massive, unclear line chart?”
• Valve's "Solution" to Review Bombing Ignores Steam's Longstanding Problems by Patrick Klepek [Waypoint]
“Look at Firewatch's review graph, for example. For one, the graph doesn't show up on a game's Steam page by default. You need to consciously click "show graph," which means most people aren't ever going to see this. Two, the idea that a Steam user will take time out of their day to become an amateur reporter and discover the mystery behind a spike in negative user reviews is horribly naive. Even as a trained reporter, I often find myself relying on the snap judgement usefulness of an average review score on Steam to help me understand if a game is worth checking out. Review bombing relies on casual laziness to be effective; at the time the review bomb sticks, some number of users will rely on snap judgement usefulness to walk away from a game. As a platform holder, it's Valve's job to make their service more useful. Even though a casual search of Firewatch on Google would instasntly produce a bunch of articles about what's happened in the past few weeks, there's nothing on Steam itself that points you in that direction. Did a patch break the game's balance, introduce frame rate issue? Was a recent piece of downloadable content confusing? Steam could add some form of usefulness to this graph by linking highly shared articles about the game to the graph, providing context.”
posted by Fizz (28 comments total) 6 users marked this as a favorite
 
Steam Spy (website) weighs in on Twitter. Small excerpt:
"There is no correlation between review bombing titles released a while ago and decline in their sales."
posted by ODiV at 8:01 AM on September 21, 2017 [1 favorite]


That doesn't do shit. They're actually on the side of the review bombers.
posted by Artw at 8:03 AM on September 21, 2017 [6 favorites]


It sucks, because there's not a really good way to control for this kind of thing mathematically without creating potentially big sensitivity/specificity problems for reviews of games that aren't being shitbombed. But "we'll let the user do more work and evaluate more data" is never a viable solution. Especially if you make them hunt for that data.

Maybe if you detect a sudden and violent downward swing in ratings, flag it with an alert that tells the user that this could be the result of recent problems with the game, but it could also be a personal or ideological problem people have with the developer.

Podcasts suffer this too. John Gruber used to call The Talk Show "America's favorite 1-star podcast" after he pissed off half his listeners via a very badly handled break with Dan Benjamin and his network. iTunes got hammered with bad ratings. It took a long, long time for his ratings to recover, and it's been one of the most popular podcasts in the English-speaking world.

At bottom, I'm going to buy Firewatch this week even though I have almost zero time for games, because fuck those jerks. Enjoy taking your stand while I give money to the people who were mean to the gibbering rape joke bigot you worship.
posted by middleclasstool at 8:09 AM on September 21, 2017 [8 favorites]


The best way would be an algorithm that tracked score fluctuations over short periods of time (nothing wrong with a game that starts strong, but people over time start finding more and more flaws with it), and if it detected a sudden spike without any palpable reason such as a patch or DLC, would simply freeze user ratings in a pre-spike level until Valve investigated if the spike was caused by some bombing campaign. If it was some oddity, let the review slide, if it was an effect of a bombing, all users who scored or (perhaps more important) changed scores during that period would have their scores permanently erased from the averages. After a while, there would be a database of ill-intended users that could have all their user scores ghosted automatically because they'd be untrustworthy anyway.

Of course, that would imply Valve to care.
posted by lmfsilva at 8:30 AM on September 21, 2017 [2 favorites]


Of course, that would imply Valve to care.

Valve has created a money printing machine, and the reason they call it "Fuck you money" is because they don't have to care.

Reviews on the internet == Comments on the internet, and really, don't read the comments. Wait for the game to go on sale if you're unsure, and if it sucks, well, so what. Even my favorite restaurant sometimes drops the ball.
posted by Pogo_Fuzzybutt at 8:35 AM on September 21, 2017 [1 favorite]


I think we need to admit, as a society, that "algorithms" don't solve anything. Actual human beings need to look at reviews and pull the low-quality ones. And you can't fucking crowd-source it, you need to have a specific group of people who are trained on metrics of quality and whose decisions are reviewed by other people. Editors, basically. That's the only true solution to this problem.

Since nobody wants to spend the money on a real solution, though, giving end users more information is helpful. Though I'd argue that the best no-money-needed solution would be to give end uses less information. Pull the user reviews entirely, let the games sink or swim based on professional reviewers and gameplay streams. There are lots of ways to manipulate those things as well, sure, but there are few situations that are actually improved by mob rule.
posted by tobascodagama at 8:37 AM on September 21, 2017 [9 favorites]


Reviews on the internet == Comments on the internet, and really, don't read the comments.

Well that makes it easy. Unmoderated comments sections are cesspools and should universally be shut down.
posted by Artw at 8:43 AM on September 21, 2017 [2 favorites]


Reviews on the internet == Comments on the internet, and really, don't read the comments. Wait for the game to go on sale if you're unsure, and if it sucks, well, so what.

The issue is that these trolls have so infected gaming culture and video games that they've basically weaponized comments. These things impact how video games are made, how players engage and play the game, and communicate with the developers.

Valve's solution here seems like a very small drop in a much larger piss bucket. You're right, reading those comments is basically hell. But to just ignore the issue is not the solution.
posted by Fizz at 8:45 AM on September 21, 2017 [3 favorites]


Every "product reviewing" system on the internet is subject to Being Gamed. And Valve sells Games to Gamers, so it just goes double/triple/nth-le for them.
posted by oneswellfoop at 8:55 AM on September 21, 2017 [4 favorites]


Code is never neutral, as much as its proponents want it to be. By abstracting the data to a histogram that nobody will pay attention to, they've decided that the shitty reviews will stay.

It's their platform. They could remove shitty/disruptive voices from these asshats if they wanted to, but... filthy lucre.
posted by fifteen schnitzengruben is my limit at 9:14 AM on September 21, 2017 [2 favorites]


Funny, this is more or less the same thing the US federal government agencies do (internally) in response to online petitions for notice and comment in governance.

So, I think Valve have probably co located a locally optimal solution.

where the locale is not really caring.

people in a polity can get a lawyer and sue the government, not sure what lowly consumers can do against a Producerist corporation.

Some US agencies have moved away from hearings (which guarantee contention) and toward "open houses" and "Q and As", which are more service oriented, and have give and take. perhaps this innovation can spread to the private sector.
posted by eustatic at 9:15 AM on September 21, 2017 [1 favorite]


Jeez, Valve- stop trying to make me actually like Origin.
posted by TheWhiteSkull at 9:24 AM on September 21, 2017 [1 favorite]


Of course, that would imply Valve to care.

One of the surprising things in Zoe Quinn's book is the disclosure that Valve gives basically zero shits about community harassment and has nobody whose job it is to deal with it. So this, which is yet another tech industry attempt to evade the responsibility to do the work, is not a surprise. It's completely the sort of thing that can be done quickly, tossed out to the public, and then they can go back to doing whatever they actually feel like doing.
posted by Pope Guilty at 9:46 AM on September 21, 2017 [7 favorites]


There are ways to make it useful, if they're so inclined:

First and foremost, require a verified purchase. Why the hell should anyone be able to review a game if they haven't actually played it?

Beyond that, use machine learning and algorithms to determine if someone's review is constructive. We can analyze text to determine its reading level, the level of vitriol, whether it's plagiarized, et cetera. If a review is a copypasta, a rage-dump, or other unconstructive rant, or appears to be about something other than the game itself, it should be flagged as "unconstructive."

Weight the review average by how constructive the reviews are. 5-star reviews with no comments are just about as unhelpful as 1 star reviews that say "Zoe Quinn is a ****." This has a side effect of encouraging users to actually write decent reviews, even if they want to criticize a game. But it absolutely sets a higher bar for participation in a review bomb, positive or negative.
posted by explosion at 9:55 AM on September 21, 2017 [1 favorite]


On the one hand, "Be good consumers and use these tools we're giving you for 'feedback' only in the service of giving us more money," sucks, but on the other hand, GamerGaters and racists suck more.

Why the hell should anyone be able to review a game if they haven't actually played it?

They can't.

Beyond that, use machine learning and algorithms to determine if someone's review is constructive. We can analyze text to determine its reading level, the level of vitriol, whether it's plagiarized, et cetera.

Dammit, I can't find it right now, but there was that AI thing going around recently that tried to determine if a statement was offensive or problematic. "I am a straight white man." was found to have a low chance of offense. "I am a black lesbian woman." had a high chance. I'm going to go looking for it because it was interesting, but I don't think this is easy stuff.
posted by ODiV at 10:04 AM on September 21, 2017 [3 favorites]


First and foremost, require a verified purchase. Why the hell should anyone be able to review a game if they haven't actually played it?

Steam already does this. However, you're allowed to return a game for a full refund if you've played it for <2 hours and owned it for <2 weeks. So review-bombers can buy a game, leave a review, then get their money back. Technically Valve can revoke refund privileges for an account that abuses them, but that doesn't actually seem to happen. (Again, that would require intervention by a human being.) Deleting the review after a refund isn't a great solution either: if a game has enough genuine problems to be worth returning, you shouldn't need to decide between warning other users and getting your money back.
posted by skymt at 10:17 AM on September 21, 2017 [5 favorites]


Reviews on the internet == Comments on the internet, and really, don't read the comments.

Well that makes it easy. Unmoderated comments sections are cesspools and should universally be shut down.


Cortex has mentioned bringing games to FanFare. Should be fun!
posted by adept256 at 10:29 AM on September 21, 2017


"Ultimately, the decision was to do nothing..." which is, of course, a best practice in all industries.
posted by Billiken at 10:29 AM on September 21, 2017 [4 favorites]


"Unmoderated" is a very important distinction there.
posted by Artw at 10:36 AM on September 21, 2017 [2 favorites]


There's no way their reviews are unmoderated. Try selling drugs or phishing for passwords with them and see how long that lasts. Valve's just showing us their priorities.
posted by ODiV at 10:40 AM on September 21, 2017 [8 favorites]


"There is no correlation between review bombing titles released a while ago and decline in their sales."

Yeah, for me to care about this, I don't want a fancy graph of review scores over time. I need data that demonstrates steam user reviews have any causal effect on sales whatsoever, which seems like a very dubious hypothesis.
posted by straight at 11:26 AM on September 21, 2017 [1 favorite]


I love the idea of time charts.

I buy indie games, and weird tiny publisher games (I'm not sure where the line is), and oddball genre games (hidden object, point-and-click adventures, jigsaw puzzles), and overall review rating tells me very little - there are awesome games that have "mixed" reviews because they went on sale, and a whole bunch of people who hate the genre tried them and posted reviews that say, basically, "this game sucks because it works like all the other games in its genre." Charts will let me see if the negative reviews cluster around seasonal sales, or are spread throughout the life of the game.

I have very little sympathy for people who intend to spend $30-$80 on a game but can't be bothered to skim the text of the reviews to find out if they're "this game is terrible and it sucks" vs "this game has slow pacing and poor voice acting and no accuracy for gunshots and there's a bug that can strip your entire inventory."

Why the hell should anyone be able to review a game if they haven't actually played it?
> They can't.


They can't review a game without a Steam key, and Steam has already taken action to sort between buyers-on-Steam vs people who acquired a key elsewhere - there were devs giving away keys for the promise of good reviews, and Steam adjusted its calculations so those no longer have the same weight in the overall rating. This does mean that buying a game at Humble Bundle or similar won't let you post a weighted review, but you're still able to vote Up/Down and make comments.

However: You don't have to have "played" a game to review it. You have to have it in your Steam library. This includes family/friend borrowed games, and keys acquired from various other sites. You may have had to actually open the game to post a review, but there's definitely no time limit for play required.
posted by ErisLordFreedom at 3:32 PM on September 21, 2017 [3 favorites]


but there's definitely no time limit for play required.

Yes there is, but it's super short. I'm not sure there's a great way to make sure people have played it no matter how long the required time is anyway. So you're right in that you can indeed review a game without having played it. I'm not sure if it's really possible to enforce that sort of thing though?
posted by ODiV at 12:01 AM on September 22, 2017


I know that I personally pay attention to the number of hours listed next to a reviewer. If I see them complaining about something and there's only .5 hours of game play...I'll temper my opinion of their commentary. People are of course entitled to offer their opinions, but I'm going to reconsider how much weight I give those opinions based on other factors.
posted by Fizz at 9:20 AM on September 22, 2017


The only solution to "didn't play it much; left bad review" is for other players to read the reviews. There's a difference between a five-minutes-played review that says "this sucks and it's terrible; sending for refund" and one that says "crashed four times trying to get it started, hung up on the character selection screen, froze my entire computer until I force-ended the program; sending for refund." (Same problem goes for spotting good reviews paid for by giving out free keys: "yay I love this" is different from "the acting is terrific and the puzzles were challenging but not impossible.")

If extra time were added to the requirements, swarm-reviews would just leave the game on in the background to hit that time level. If that time level were beyond where a refund is possible, that means people who honestly found a game so terrible they got a refund, wouldn't be able to leave reviews.
posted by ErisLordFreedom at 12:37 PM on September 22, 2017


Ok, so, the thing is that AAA games aren't really hurt by review bombs. They're not even hurt by low MetaCritic scores from professional reviews. (With the obvious exception of the totally gross thing where publishers/studios make bonuses contingent on getting a Metacritic score about 80 or whatever.) But basically nobody is making a purchase decision on the new Fallout game based on an aggregate review score.

However, the aggregate review score is extremely important for smaller games that count on Steam's "discovery" features because they can't afford traditional marketing. When you're flipping through your discovery queue, you're not going to read reviews on a "mixed" game, you're just going to click through to the next one.

If the review histograms are easily visible at a first glance on a product page, this could be a huge benefit for indie games that come under attack by review bombs.
posted by tobascodagama at 1:34 PM on September 22, 2017


However, the aggregate review score is extremely important for smaller games that count on Steam's "discovery" features because they can't afford traditional marketing.

That seems like a very difficult hypothesis to test since in an ideal world both review scores and sales would be caused by the quality of the game rather than sales being caused by review scores. I doubt Valve even has sufficient data to figure out what if any effect player review scores have on sales.
posted by straight at 1:53 PM on September 23, 2017


I suppose if they wanted hard data, they'd have to do an A / B test where half the users see a different review score than the other half.

Though I think simply appearing on the "discovery" lists will be much more significant than what rating is shown. If the review-bombing crowd shows certain similarities of interest, which I expect they will, then the attempts at personalizing the discovery lists could limit the impact to their own demographic.
posted by RobotHero at 9:30 AM on September 24, 2017 [1 favorite]


« Older Access denied: wheelchair metro maps versus...   |   Angry about "Sandwich au fromage fondant"? Send a... Newer »


This thread has been archived and is closed to new comments