Greetings!
In the Shadowdark game thread, GeekyBugle and Rhialto brought up discussing RPG reviews and so on. So, I thought I would create a discussion thread where RPG reviews, standards, and expectations can be discussed, without derailing other discussing threads.
What do you think should be the standard for RPG reviews? Or is there no "one standard"? To whit, many people, whether creators, designers, or gamers themselves all have different standards, different policies, and oftentimes, entirely different expectations.
Semper Fidelis,
SHARK
IMHO it should be standards that promote transparency and confidence:
Full disclosure of any connection with the producers of the product.
Full disclosure of whether the product was bought or not.
Full disclosure of whether ANY payment for the review was made.
There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.
IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.
Greetings!
As for my opening discussion prompt, I am quite familiar with heretofore proclaimed and established professional academic review standards and expectations. When it comes to games, I have never held game reviews to the same kind of professional academic review standards. Professional academic review standards are focused with presentation, discussion, of truth, of evidence, historical and archeological sources. Real data. Of course, all of this has immense implications--a particular scholar reviewing various history books for example, must adhere to strict standards and policies--otherwise, their career as a reviewer is simply over with. Likewise, a Historian writing a History book must also exercise care in adhering to and observing essential academic standards and expectations. Promoting nonsense and BS will quickly end a scholar's or a Historian's career.
Games, for the most part, and especially RPG's, are virtually entirely subjective. Then, you have the actual practices of innumerable "reviewers" and the mutable and subjective expectations of the gamers themselves. Therefore, I, personally, do not have super-high expectations for "game reviews" from anyone. I just want a look at the actual product, it's layout, components, and overview of what the product presents. The reviewer's "objectivity" or their opinion, per se, is virtually meaningless. It doesn't matter to me if some reviewer was paid for example. Or receive free goodies. I assume they all do, as I know such are standard practices or many throughout RPG's, but also miniatures, and hobby paints, as well as hobby terrain. It all comes own--mostly--to subjective, person opinion. A minimum actually deals with what the product physically presents. Otherwise, it is all down to does the reviewer like the particular product, or not?
What do you think?
Semper Fidelis,
SHARK
Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:
Full disclosure of any connection with the producers of the product.
Full disclosure of whether the product was bought or not.
Full disclosure of whether ANY payment for the review was made.
There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.
IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.
I find it very odd that people would be paid money for a review in this sphere instead of getting a free copy of the book/pdf. How common does this even happen?
Thanks Shark.
I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.
In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.
But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.
Quote from: rhialto on March 09, 2023, 04:38:58 PM
Thanks Shark.
I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.
In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.
But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.
It's not about QB making anyone buy a product or not, it's about the getekeeping involved, the incestuous relationships of "reviewing" a product you worked on (without disclosure) or that you're friends with the creator, etc.
Dishonest practices will subvert a culture of meritocracy by propping up their friends.
Some people are very transparent about everything that could influence their review, which makes them more creditable on an initial glance. Other people are obviously snake-oil salesmen, to the point I'd look out the window to verify it was raining if they insisted that it was. Most people are going to be somewhere in the mushy middle--if for no other reason, it's not always easy to see what viewers would consider an influence.
Then there's pure click-bait. The reviewer's credibility isn't even the problem when we get to that stage. It takes work and skill to be a thoughtful critic. The click-bait artist isn't even beginning to pretend to try anymore. Good, thoughtful criticism doesn't appear full-blown from the head of Zeus. As long as someone is at least trying, I'll cut them some slack on that front. If they ever stop trying, or it's obvious that their tastes are so different than mine and so ingrained that they can't review from the perspective of other people, I don't hold it against them. I just ignore them.
Quote from: GeekyBugle on March 09, 2023, 04:45:01 PM
It's not about QB making anyone buy a product or not, it's about the getekeeping involved, the incestuous relationships of "reviewing" a product you worked on (without disclosure) or that you're friends with the creator, etc.
Dishonest practices will subvert a culture of meritocracy by propping up their friends.
But you have total control over that: maybe refuse to buy anything that doesn't have a free version? Or make yourself immune to such dishonest practitioners, by refusing to buy products recommended by reviewers who don't disclose their standards? And even in the case where someone does disclose their standards you're taking them at their word, which always has some uncertainty.
It should be apparent now that I just don't buy any game-related material anymore which doesn't have a free version to evaluate myself, or is not from a creator I've come to value. And in the latter case it can be hit or miss, like anything else.
Quote from: rhialto on March 09, 2023, 04:38:58 PM
Thanks Shark.
I'll restate my earlier position: in cases where a free version is available for one to make up one's own mind, other reviews should be ignored in most cases. And creators who provide such free versions should be lauded.
In cases where there is no free version to check out I'm all for reviewers conducting themselves as they see fit: paid, unpaid, whatever. The market will sort out who's a reliable reviewer and who isn't, for each of us. Example: I watch QB review a game, I buy it, read it and my opinion of the value is wildly different from his. Now I know not to rely on QB for matching my opinion. Or the reverse happens and it's a 100% match, and now I know I can rely on QB for opinions matching my own. Obviously with some uncertainty factor.
But in all cases: caveat emptor. You decided to buy it, you're responsible for that decision, not some mindflayer influencer.
Greetings!
You are welcome, Rhialto!
I agree. I know from experience, for example, of observed tastes and game experience, that I like much of what Professor Dungeon Master of Dungeoncraft talks about. He often reviews games that interest him, and that he likes. So, I take that into consideration. I know PDM is a big fan of grittier, OSR games. Thus, I weight his assessments based on that.
But as you say, I'm the customer that actually chooses to spend my money on product X, so caveat emptor! What I think of a product is far more important than what any one game reviewer thinks, for example.
In a slightly related field, I did the same thing with people's reviews of the video game, of HOIIV, "No Step Back". I read numerous customer reviews, and I listened to half a dozen influencer's reviews/Content Creators/Proessional Gamers. When enough of that lined up--and the discussion evalluated gameplay, and particur game mechanics--with no probems in my opinion--I then pulled the trigger and bought the game expansion. Video gmes have more objective stuff in them, but they are still largely based on styistic choices, and subjective opinions, experiences, and preferences.
I tend to view RPG reviews the same way--with lots of salt, and simply looking for overlap between gamers I know of, and myself. As concerning similar preferences, because there is only a minimal degree of objective truth in RPG's or video games. Most of it all comes down to personal opinion and preferences.
Semper Fidelis,
SHARK
Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:
Full disclosure of any connection with the producers of the product.
Full disclosure of whether the product was bought or not.
Full disclosure of whether ANY payment for the review was made.
Quote from: Steven Mitchell on March 09, 2023, 04:54:19 PM
Some people are very transparent about everything that could influence their review, which makes them more creditable on an initial glance. Other people are obviously snake-oil salesmen, to the point I'd look out the window to verify it was raining if they insisted that it was. Most people are going to be somewhere in the mushy middle--if for no other reason, it's not always easy to see what viewers would consider an influence.
I think it's a reasonable ask to have a disclaimer revealing potential bias. Pundit has done that for a number of print reviews -- like when it is a product of a business partner, or a product that he has worked on. Questing Beast has done the same by explicitly saying when something is a sponsored review. These both seem fine to me, and I think they match GeekyBugle's ask. We should encourage more such disclaimers.
On the other hand, I also don't think it's a big deal. Like others have said, I don't really care about what the reviewer thinks of the product as a whole -- because especially with TTRPGs, everything is hugely subjective. What I care about is facts that the reviewer reveals -- what are the mechanics like, what characters are possible, what adventure material is supplied, etc.
Quote from: jhkim on March 09, 2023, 05:05:38 PM
On the other hand, I also don't think it's a big deal. Like others have said, I don't really care about what the reviewer thinks of the product as a whole -- because especially with TTRPGs, everything is hugely subjective. What I care about is facts that the reviewer reveals -- what are the mechanics like, what characters are possible, what adventure material is supplied, etc.
I agree with that mostly, up to a point. It's possible to use "selective omission" to make something seem different and/or better than it is. I could, for example, write a review of AD&D 1E that makes it sounds a lot simpler than it is. It's a little trickier, and I'd have to be skating up to the edge of being dishonest, but I could make the D&D Rules Cyclopedia sound much more complex than it is. (No, I'm not going to prove that. It would be a whole lot of work to no useful purpose.)
But yeah, normally it doesn't matter. When someone gets on a video and chats about a game as a "review", it's not that organized, typically.
I only want to know if they've been paid or not, or if they are somehow connected to the title through work or investure. Seriously, who's going to be be objective about a kickstarter they wrote for or bought the top tier so they could get the limited edition leatherette (unless they supported it after they reviewed it).
For me, it boils down to the reviewer's reputation.
There's was a fellow on drivethru named Thilo Graf that wrote hundreds and hundreds of in-depth reviews. Patrons provided the titles he reviewed in either pdf or dead tree. Unlike most of the 'Featured' reviewers on drivethru who only give 5 star 'reviews', he actually handed out 1 and 2 star reviews.
I don't know anything about the guy personally, but he always gave me the impression he was impartial and not bought and paid for.
It's curious to note that his last review was on "Blood in the Chocolate" in Nov 2021 and there's been nothing since.
Someone else mentioned in another thread Seth Skorkowsky, also a perfect example of a good reputation. Granted, none of the titles he reviews are stinkers, so no 1 stars there. But, I view that as he has read plenty of throwaways, and what he gives us the cream of the crop, the gems we may have missed.
TLDR: Infomercials = bad.
Quote from: GeekyBugle on March 09, 2023, 04:30:45 PM
IMHO it should be standards that promote transparency and confidence:
Full disclosure of any connection with the producers of the product.
Full disclosure of whether the product was bought or not.
Full disclosure of whether ANY payment for the review was made.
There's of course no way to enforce this, and I'm not sure I would want one. But it is a discussion that was started by GamerGate and then placated by empty gestures.
IMHO the public, as consumers of both the reviews and the products are entitled to this transparency. Honest reviewers and creators are entitled to raise concerns about it and to even criticize those who don't adhere to these standards.
Yes, this. This is the minimum. Any less should tarnish your reputation.
I do all sorts of reviews: for friends, for people that have asked, for adventures I read and others I actually played.
Always with a disclaimer explaining the situation.
Disclaimers are good, but what rhialto says about taste is probably more important. But I think details are more important than that. A particular reviewer might have quite different tastes from mine, but the information they give is still valuable. They could address something they don't like, and describe it in sufficient detail that I decide I would like it for the exact same reason they don't. So concrete examples are important. In a video format it's nice if they show some pages from the book or PDF. In a written format it's nice if they provide some direct quotes. Actual play experience is also nice. Say what went well, what went wrong. What was confusing. That's something I can relate to my own experiences, or perhaps just be prepared for it so the same mistake doesn't happen if I like it and get the opportunity to play in or run a game with it.
If you have any kind of tie to a person who was involved in the creation of the thing you are reviewing, you recuse yourself. You DO NOT review it. End of. You say you won't review it because of a conflict of interest. You disclose that conflict of interest and then move on. It doesn't matter if the product is good or bad. You have a bias in favour the people who made the product. That alone should be a breach of trust. The one and only exception would be receiving a product for free to review. But that should be a matter of course as a reviewer and not a privilege granted only to those the product maker likes.
Quote from: Jason Coplen on March 10, 2023, 01:37:15 PM
Quote from: GeekyBugle on March 09, 2023, 04:17:05 PM
Quote from: rhialto on March 09, 2023, 04:08:51 PM
Quote from: GeekyBugle on March 09, 2023, 03:59:00 PM
The real discussion should be about the reviewers, the game is just the catalyst, the 5e players that MIGHT play it are irrelevant.
IMHO the discussion should be centered around the practices on reviewing RPGs.
I think this is true only in the cases where no freely available version is offered, to review oneself. Anyone else's opinion in such cases where a free version is available is...inconsequential? Irrelevant? So, in the case of Shadowdark: if someone is not availing themselves of the quickstart, and only relying on others' opinions, that's their choice, and no one else should care. Maybe a separate thread on reviewers is warranted.
For instance, getting a free copy (when no one else is getting it) of the whole printed book is common practice. Should it be? Wouldn't buying the book be more impartial?
I don't know, but I do know buying your own copy is an investment. Maybe you'd think more of the game, or less in the case it blew goats nuts. Then you'd feel cheated out of x dollars.
Someone mentioned (here I think) a reviewer that buys the books with money he gets from his supporters, that's one way to solve it.
IF I was dedicated to doing reviews I wouldn't feel cheated from X Dollars by buying a product I didn't like, it generates content and negative content drives more clicks than positive content.
Quote from: SHARK on March 09, 2023, 04:18:08 PM
What do you think should be the standard for RPG reviews? Or is there no "one standard"? To whit, many people, whether creators, designers, or gamers themselves all have different standards, different policies, and oftentimes, entirely different expectations.
Semper Fidelis,
SHARK
I don't know that there is one standard but I like to see people disclose business or personal relationships in reviews, and mention if there was any comp supplied (I've had a review where I sent PDFs and they even labeled it sponsored content, which I think is fair, but just specifying if a comp was given and in what format is probably useful). I would also say when reviewers explain their review criteria that is good (and it can also be handy if they give the style of gaming they come from or explain what biases in terms of system and setting they might have)
At the same time, a lot of this is just someone informally giving their thoughts on a blog or something so I think you have to take it case by case (the more seriously one presents themselves as a reviewer, the more I would expect higher standards).
Quote from: King Tyranno on March 09, 2023, 07:36:05 PM
If you have any kind of tie to a person who was involved in the creation of the thing you are reviewing, you recuse yourself. You DO NOT review it. End of. You say you won't review it because of a conflict of interest. You disclose that conflict of interest and then move on. It doesn't matter if the product is good or bad. You have a bias in favour the people who made the product. That alone should be a breach of trust. The one and only exception would be receiving a product for free to review. But that should be a matter of course as a reviewer and not a privilege granted only to those the product maker likes.
Quote from: Bedrockbrendan on March 10, 2023, 01:56:21 PM
I don't know that there is one standard but I like to see people disclose business or personal relationships in reviews, and mention if there was any comp supplied (I've had a review where I sent PDFs and they even labeled it sponsored content, which I think is fair, but just specifying if a comp was given and in what format is probably useful). I would also say when reviewers explain their review criteria that is good (and it can also be handy if they give the style of gaming they come from or explain what biases in terms of system and setting they might have)
At the same time, a lot of this is just someone informally giving their thoughts on a blog or something so I think you have to take it case by case (the more seriously one presents themselves as a reviewer, the more I would expect higher standards).
Yeah. I think King Tyranno is treating RPG reviews as something weighty and serious like a court case, when it's more often an off-the-cuff thoughts on a product. This is especially because RPGs are a huge time investment that reviewers often don't have time to actually play through a game book. So it's really just a look-through, and the important part is the description more than the assessment.
I think disclaimer for bias is fine. I generally assume that a reviewer hasn't played through the product. If they have, that should be mentioned. For example, here's Pundit's disclaimer for his review of Raiders of R'lyeh.
QuoteI'll repeat what I said in the previous review: I was a major consultant on this game, and so it has some of my own work in it. I was (well) paid to consult, but I do not make any money from sales of the product now. I obviously have a favorable bias about Raiders due to my involvement with it, and I wanted to make that transparent; but I do think I'll still be able to be fair and accurate about the content.
http://therpgpundit.blogspot.com/2018/04/rpgpundit-reviews-raiders-of-rlyeh_25.html
He's quite clear about it, and I think it's fine.
I think reviews should contain these elements.
First should be the cost of the product in terms of cash and time investment to get use out of it.
Eg: a 32 page booklet will be cheaper and quicker to learn than a 320 page omnibus. Is the product lore heavy, crunchy or compatible with existing games?
Art quality and feel along with genre should also be discussed. This might need to be first. In addition to genre, playstyle or target audience would be nice to know. ( OSR, modern, something else) I wouldn't bother with reading a review of Thirsty Sword Lesbians even if the tile was more ambiguous like LofFP.
The build quality, editing and layout also need to be touched on. I think those are the bare bones of a review that would be useful.
Quote from: GeekyBugle on March 10, 2023, 01:48:46 PM
Someone mentioned (here I think) a reviewer that buys the books with money he gets from his supporters, that's one way to solve it.
I would wonder as a potential supporter where I actually get value for my money over just buying the book myself, reading through it, and uploading my own review.
The reviewer would need to be regularly reviewing products I'm interested in, and not reviewing too many products I wouldn't buy anyway, with a fairly low support tier for that to pay off.
The key for me is disclosure and transparency. Like I said elsewhere, just give me the facts. I am smart enough to factor it in or out.
Personally, if I did reviews on a YouTube channel and a friend of mine created something, I would tell everybody about the relationship, and purposefully call it a preview instead of a review.