Greetings!

In the Shadowdark game thread, GeekyBugle and Rhialto brought up discussing RPG reviews and so on. So, I thought I would create a discussion thread where RPG reviews, standards, and expectations can be discussed, without derailing other discussing threads.

What do you think should be the standard for RPG reviews? Or is there no "one standard"? To whit, many people, whether creators, designers, or gamers themselves all have different standards, different policies, and oftentimes, entirely different expectations.

Semper Fidelis,

SHARK

IMHO it should be standards that promote transparency and confidence:

Full disclosure of any connection with the producers of the product.

Full disclosure of whether the product was bought or not.

Full disclosure of whether ANY payment for the review was made.

There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.

IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.

Greetings!

As for my opening discussion prompt, I am quite familiar with heretofore proclaimed and established professional academic review standards and expectations. When it comes to games, I have never held game reviews to the same kind of professional academic review standards. Professional academic review standards are focused with presentation, discussion, of truth, of evidence, historical and archeological sources. Real data. Of course, all of this has immense implications--a particular scholar reviewing various history books for example, must adhere to strict standards and policies--otherwise, their career as a reviewer is simply over with. Likewise, a Historian writing a History book must also exercise care in adhering to and observing essential academic standards and expectations. Promoting nonsense and BS will quickly end a scholar's or a Historian's career.

Games, for the most part, and especially RPG's, are virtually entirely subjective. Then, you have the actual practices of innumerable "reviewers" and the mutable and subjective expectations of the gamers themselves. Therefore, I, personally, do not have super-high expectations for "game reviews" from anyone. I just want a look at the actual product, it's layout, components, and overview of what the product presents. The reviewer's "objectivity" or their opinion, per se, is virtually meaningless. It doesn't matter to me if some reviewer was paid for example. Or receive free goodies. I assume they all do, as I know such are standard practices or many throughout RPG's, but also miniatures, and hobby paints, as well as hobby terrain. It all comes own--mostly--to subjective, person opinion. A minimum actually deals with what the product physically presents. Otherwise, it is all down to does the reviewer like the particular product, or not?

What do you think?

Semper Fidelis,

SHARK

Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:

Full disclosure of any connection with the producers of the product.

Full disclosure of whether the product was bought or not.

Full disclosure of whether ANY payment for the review was made.

There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.

IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.

I find it very odd that people would be paid money for a review in this sphere instead of getting a free copy of the book/pdf. How common does this even happen?

Thanks Shark.

I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.

In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.

But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.

Quote from: rhialto on March 09, 2023, 04:38:58 PM
Thanks Shark.

I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.

In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.

But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.

It's not about QB making anyone buy a product or not, it's about the getekeeping involved, the incestuous relationships of "reviewing" a product you worked on (without disclosure) or that you're friends with the creator, etc.

Dishonest practices will subvert a culture of meritocracy by propping up their friends.

Some people are very transparent about everything that could influence their review, which makes them more creditable on an initial glance.  Other people are obviously snake-oil salesmen, to the point I'd look out the window to verify it was raining if they insisted that it was.  Most people are going to be somewhere in the mushy middle--if for no other reason, it's not always easy to see what viewers would consider an influence.

Then there's pure click-bait. The reviewer's credibility isn't even the problem when we get to that stage.  It takes work and skill to be a thoughtful critic.  The click-bait artist isn't even beginning to pretend to try anymore.   Good, thoughtful criticism doesn't appear full-blown from the head of Zeus.  As long as someone is at least trying, I'll cut them some slack on that front.  If they ever stop trying, or it's obvious that their tastes are so different than mine and so ingrained that they can't review from the perspective of other people, I don't hold it against them.  I just ignore them.

Quote from: GeekyBugle on March 09, 2023, 04:45:01 PM
It's not about QB making anyone buy a product or not, it's about the getekeeping involved, the incestuous relationships of "reviewing" a product you worked on (without disclosure) or that you're friends with the creator, etc.

Dishonest practices will subvert a culture of meritocracy by propping up their friends.

But you have total control over that: maybe refuse to buy anything that doesn't have a free version? Or make yourself immune to such dishonest practitioners, by refusing to buy products recommended by reviewers who don't disclose their standards? And even in the case where someone does disclose their standards you're taking them at their word, which always has some uncertainty.

It should be apparent now that I just don't buy any game-related material anymore which doesn't have a free version to evaluate myself, or is not from a creator I've come to value. And in the latter case it can be hit or miss, like anything else.

Quote from: rhialto on March 09, 2023, 04:38:58 PM
Thanks Shark.

I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.

In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.

But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.

Greetings!

You are welcome, Rhialto!

I agree. I know from experience, for example, of observed tastes and game experience, that I like much of what Professor Dungeon Master of Dungeoncraft talks about. He often reviews games that interest him, and that he likes. So, I take that into consideration. I know PDM is a big fan of grittier, OSR games. Thus, I weight his assessments based on that.

But as you say, I'm the customer that actually chooses to spend my money on product X, so caveat emptor! What I think of a product is far more important than what any one game reviewer thinks, for example.

In a slightly related field, I did the same thing with people's reviews of the video game, of HOIIV, "No Step Back". I read numerous customer reviews, and I listened to half a dozen influencer's reviews/Content Creators/Proessional Gamers. When enough of that lined up--and the discussion evalluated gameplay, and particur game mechanics--with no probems in my opinion--I then pulled the trigger and bought the game expansion. Video gmes have more objective stuff in them, but they are still largely based on styistic choices, and subjective opinions, experiences, and preferences.

I tend to view RPG reviews the same way--with lots of salt, and simply looking for overlap between gamers I know of, and myself. As concerning similar preferences, because there is only a minimal degree of objective truth in RPG's or video games. Most of it all comes down to personal opinion and preferences.

Semper Fidelis,

SHARK

Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:

Full disclosure of any connection with the producers of the product.

Full disclosure of whether the product was bought or not.

Full disclosure of whether ANY payment for the review was made.

Quote from: Steven Mitchell on March 09, 2023, 04:54:19 PM
Some people are very transparent about everything that could influence their review, which makes them more creditable on an initial glance.  Other people are obviously snake-oil salesmen, to the point I'd look out the window to verify it was raining if they insisted that it was.  Most people are going to be somewhere in the mushy middle--if for no other reason, it's not always easy to see what viewers would consider an influence.

I think it's a reasonable ask to have a disclaimer revealing potential bias. Pundit has done that for a number of print reviews -- like when it is a product of a business partner, or a product that he has worked on. Questing Beast has done the same by explicitly saying when something is a sponsored review. These both seem fine to me, and I think they match GeekyBugle's ask. We should encourage more such disclaimers.

On the other hand, I also don't think it's a big deal. Like others have said, I don't really care about what the reviewer thinks of the product as a whole -- because especially with TTRPGs, everything is hugely subjective. What I care about is facts that the reviewer reveals -- what are the mechanics like, what characters are possible, what adventure material is supplied, etc.

Quote from: jhkim on March 09, 2023, 05:05:38 PM

On the other hand, I also don't think it's a big deal. Like others have said, I don't really care about what the reviewer thinks of the product as a whole -- because especially with TTRPGs, everything is hugely subjective. What I care about is facts that the reviewer reveals -- what are the mechanics like, what characters are possible, what adventure material is supplied, etc.

I agree with that mostly, up to a point.  It's possible to use "selective omission" to make something seem different and/or better than it is. I could, for example, write a review of AD&D 1E that makes it sounds a lot simpler than it is.  It's a little trickier, and I'd have to be skating up to the edge of being dishonest, but I could make the D&D Rules Cyclopedia sound much more complex than it is.  (No, I'm not going to prove that.  It would be a whole lot of work to no useful purpose.)

But yeah, normally it doesn't matter.  When someone gets on a video and chats about a game as a "review", it's not that organized, typically. 

I only want to know if they've been paid or not, or if they are somehow connected to the title through work or investure. Seriously, who's going to be be objective about a kickstarter they wrote for or bought the top tier so they could get the limited edition leatherette (unless they supported it after they reviewed it).

For me, it boils down to the reviewer's reputation.

There's was a fellow on drivethru named Thilo Graf that wrote hundreds and hundreds of in-depth reviews. Patrons provided the titles he reviewed in either pdf or dead tree. Unlike most of the 'Featured' reviewers on drivethru who only give 5 star 'reviews', he actually handed out 1 and 2 star reviews.

I don't know anything about the guy personally, but he always gave me the impression he was impartial and not bought and paid for.

It's curious to note that his last review was on "Blood in the Chocolate" in Nov 2021 and there's been nothing since.

Someone else mentioned in another thread Seth Skorkowsky, also a perfect example of a good reputation. Granted, none of the titles he reviews are stinkers, so no 1 stars there. But, I view that as he has read plenty of throwaways, and what he gives us the cream of the crop, the gems we may have missed.

TLDR: Infomercials = bad.

Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:

Full disclosure of any connection with the producers of the product.

Full disclosure of whether the product was bought or not.

Full disclosure of whether ANY payment for the review was made.

There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.

IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.

Yes, this. This is the minimum. Any less should tarnish your reputation.

I do all sorts of reviews: for friends, for people that have asked, for adventures I read and others I actually played.

Always with a disclaimer explaining the situation.

Disclaimers are good, but what rhialto says about taste is probably more important. But I think details are more important than that. A particular reviewer might have quite different tastes from mine, but the information they give is still valuable. They could address something they don't like, and describe it in sufficient detail that I decide I would like it for the exact same reason they don't. So concrete examples are important. In a video format it's nice if they show some pages from the book or PDF. In a written format it's nice if they provide some direct quotes. Actual play experience is also nice. Say what went well, what went wrong. What was confusing. That's something I can relate to my own experiences, or perhaps just be prepared for it so the same mistake doesn't happen if I like it and get the opportunity to play in or run a game with it.

If you have any kind of tie to a person who was involved in the creation of the thing you are reviewing, you recuse yourself. You DO NOT review it. End of. You say you won't review it because of a conflict of interest. You disclose that conflict of interest and then move on. It doesn't matter if the product is good or bad. You have a bias in favour the people who made the product. That alone should be a breach of trust. The one and only exception would be receiving a product for free to review. But that should be a matter of course as a reviewer and not a privilege granted only to those the product maker likes.