So many criteria to judge RPG's are subjective that I got to wonder if there were any actual objective criteria for an RPG to be good. I came up with a few so far
1) The game must be fun for the target audience
2) The rules are no more complicated than they have to be to accomplish their purpose.
3) The rules are clearly written and well organized.
Are there any others? Or shoot down the ones above if you like.
Quote from: Mishihari;1121898So many criteria to judge RPG's are subjective that I got to wonder if there were any actual objective criteria for an RPG to be good. I came up with a few so far
1) The game must be fun for the target audience
2) The rules are no more complicated than they have to be to accomplish their purpose.
3) The rules are clearly written and well organized.
Are there any others? Or shoot down the ones above if you like.
Those are still all subjective.
Define first what role-play is, and how it's done by the players. Then judge RPG rules on whether they are a benefit or a hindrance to the players. Don't know if there will ever be objective role-play though.
When I roll the dice, using their rules - do the results make me either puke, nose/ear/eye-bleed, or flip the table over in rage? Those are my starting criteria.
The index in the D&D 5E Players Guide is objectively bad.
Example: Look up Temporary Hit Points
Here's the entry.
Quotetemporary hit points. See under hit points
Which leads us to...
Quotehit points
....... temporary, 198
This indexing is objectively worse than one in which they put the damn page number in both locations.
Quote from: Mishihari;1121898So many criteria to judge RPG's are subjective that I got to wonder if there were any actual objective criteria for an RPG to be good. I came up with a few so far
1) The game must be fun for the target audience
2) The rules are no more complicated than they have to be to accomplish their purpose.
3) The rules are clearly written and well organized.
Are there any others? Or shoot down the ones above if you like.
Yes, seems rather subjective, though you could apply various documentation type rules for #3. But related to #3, I can think of some positives and negatives that often come up in reviews, such as inclusions of tables of contents and indexes, grammar, display of page numbers, informative section titles, fonts & color that are easily readable, and the like.
https://www.youtube.com/watch?v=LjHORRHXtyI
There you go.
Sales
That's pretty much it for objective criteria. And even then we can discuss whether it's actually objective or not.
Rules contradict each other. If they do, that's objectively bad. Of course, the evaluation of whether or not the rules do, in fact, contradict each other, is likely to have a subjective component, because so often the intent/wording of the rules is vague.
You get the same thing with "bad math in the mechanics" and vague design intents. Read the subjective intent as generously as possible, and the math of the mechanics can't deliver, then that's objectively bad. But from a subjective premise.
A good counter example is probably the 5E "bounded accuracy" part of the design. The design intent is fairly clear (once one wades through the various places to piece it together). Whether or not it is a good thing is subjective. Whether or not they met the design is mostly objective. (There is a subjective part in defining the criteria to evaluate whether they met it or not.) That is, one could if so inclined explicitly list premises for evaluating it, explicitly reject some edge cases as irrelevant for purposes of discussion, and then make objective arguments pro or con on the mechanics meeting the design.
Maybe if you established a guild of different-minded enthusiasts, then hotly debated the various criteria penning these down into a leather-bound tome, then assembled the best 8 minds from this group to run the academy using the "Rule of 8", then had an annual awards ceremony with various categories where games would be measure by these criteria, THEN...
You might have something that stinks of objectivity. It will still be subjective, but you'll have CLOUT. Protests could be handled by maiming each loser from each category publicly. It would almost seem objective. Almost.
Quote from: trechriron;1121994Maybe if you established a guild of different-minded enthusiasts, then hotly debated the various criteria penning these down into a leather-bound tome, then assembled the best 8 minds from this group to run the academy using the "Rule of 8", then had an annual awards ceremony with various categories where games would be measure by these criteria, THEN...
You might have something that stinks of objectivity. It will still be subjective, but you'll have CLOUT. Protests could be handled by maiming each loser from each category publicly. It would almost seem objective. Almost.
Cute...
Seriously though, one reason you will never come up with an objective way to rate RPGs is that the core of what makes an RPG and RPG is actually that the games allow subjectivity as part of the game play. They aren't like chess where every attempt to move a piece is either legal or not legal. Since subjectivity is such a big part of the fun of an RPG, every person will have a different subjective reaction to the game.
So what we are left with for objectivity is objective measures of the sales of an RPG or the number of people playing it. Of course super tiny niche market Game X may objectively rate as a miserable failure because it sold 10 copies, but if there actually were only 10 people interested in the specific genre and who it is played out such that the 10 people who bought the game were the 10 people the game was targeted at, well, then you might have the MOST successful game because it hit 100% of it's market whereas by overall marketing measures D&D is generally the most successful game but it does NOT reach 100% of it's target market (some of them bought other games, or didn't buy any game at all).
Big ones for me:
-How long does combat take for me to get through? I can time a fight with a clock so this is pretty objective. Sure it varies from table to table but I don't give a shit about your table, just mine.
-This one is a bit fuzzy but "can you play without knowing the rules." There's always that one guy who never learns the rules well but you want to play with him anyway because he's your friend. Hell in my group That Guy is the host so we need to play with him. But he sucks at learning rules. With a lot of games the DM can navigate that guy through pretty easily even if the rules remain something of a black box for him, while with other games that guy just can't play. So you can't play either. Because he's your friend and you don't choose your friends based on how good they are at learning rules. He's still a great guy even if you have to remind him AGAIN what proficiency applies to even after playing the same PC in 5e for a year and half.
"Does it do what it's supposed to do?" (whatever that is)
The only question that matters.
Quote from: Daztur;1122013Big ones for me:
-How long does combat take for me to get through? I can time a fight with a clock so this is pretty objective. Sure it varies from table to table but I don't give a shit about your table, just mine.
Yea, you could have an objective measure there as long as you took care with the parameters of the measurement.
Quote-This one is a bit fuzzy but "can you play without knowing the rules." There's always that one guy who never learns the rules well but you want to play with him anyway because he's your friend. Hell in my group That Guy is the host so we need to play with him. But he sucks at learning rules. With a lot of games the DM can navigate that guy through pretty easily even if the rules remain something of a black box for him, while with other games that guy just can't play. So you can't play either. Because he's your friend and you don't choose your friends based on how good they are at learning rules. He's still a great guy even if you have to remind him AGAIN what proficiency applies to even after playing the same PC in 5e for a year and half.
Can you play without knowing the rules is going to end up being subjective I think just because it's going to rely on a subjective measure of if the player who doesn't know the rules is contributing in a way that makes you still want to play with them.
page count is objective
There have been several rpg's that I won't even look at because I know there's no damn way I'm reading a 600 page rulebook.
Quote from: goblinslayer;1122027page count is objective
There have been several rpg's that I won't even look at because I know there's no damn way I'm reading a 600 page rulebook.
Looks at Hero System.
Pretty sure there is little hope of getting agreement about what an objective measure of goodness would be.
I mean, I have long thought that White Box D&D was an utterly incomprehensible bad joke, as without knowing how to play D&D already, it was clear as mud WTF half of it was talking about. Forty years later, people hear also explained to me that I had the one printing that didn't even mention what to roll for weapon damage. However I was mightily unimpressed that the answer was that all weapons except like one magic axe do 1d6. It seems like good inspiration to make up your own rules, because you sure won't find a complete game in those three books. And yet... an large chunk of folks here will say it's a great version of D&D (though what they mean is, with added knowledge and probably various other books added, plus a good GM).
i think even sales is subjective as a measure, i've bought all kinds of material that turned out to be imho a waste of money, and many times because the guy behind the counter said "i sell a lot of these it must be good". no, rifts is half good, great setting idea, great art, didn't like the system. Vampire? just no, did not like.
For me it's got to clearly communicate what it wants to do and then mechanically do it well, the art can suck, the setting can suck, and i'd still stay its worth it. It needs to be semi-fast to learn and play, no hour or more character creation, it needs to be flexible, and i do like a small measure of simulationist take or flexibility to it, a sort of math consistency where if you know what a sword does you can scale and extrapolate to abrams tank and it still works as it should. I dislike abstraction in a game unless such is proportional to the scale; for example; abstraction in axis & allies is fine, since a a single round of rolling represents weeks of an invasion with thousands of casualties, but abstraction as you move to the individual scale becomes an irritant.
Quote from: Brad;1121939https://www.youtube.com/watch?v=LjHORRHXtyI
There you go.
O Captain, my Captain!
Quote from: goblinslayer;1122027page count is objective
There have been several rpg's that I won't even look at because I know there's no damn way I'm reading a 600 page rulebook.
But think of all the arm-workout you'll get while flexing your eye-muscles gorging on that monstrosity? You'll be JACKED once you're done.
Quote from: goblinslayer;1122027page count is objective
There have been several rpg's that I won't even look at because I know there's no damn way I'm reading a 600 page rulebook.
Hmm, but what do you count for page count? Burning Wheel Gold is almost 600 pages but a good portion of that is life paths which you don't need to absorb all of. They are also smaller pages than AD&D 1E (which clocks in at about 350 pages for the 3 core books) so isn't necessarily much different in overall size. Or how about small type like Chivalry & Sorcery 1e which was only 130 pages but due to tiny type may be close to the content of AD&D 1e.
People complain when the page count is low. People complain when the page count is high. People always complain about the price.
You don't sit down and read HERO or PF2e. You take it in bits. You start playing with the basics (I think HERO Basic is all of 98 pages?) then you dig in. The nice part is you can dig in, because there's a lot of meat on them bones.
The bigger question, the one that will help nail this conversation down nice and tight is... why do we want to come up with objective criteria for evaluating RPGs? Is the current method of evaluation not working? For example, there are TONS of free quick starts, primers and even full books available for people to kick the tires on a game...
But ultimately there are just too many differing tastes, with too many RPGs, with too many approaches to nail this down. Unless you plan on electing someone on the forum God Emperor of Tabletop Roleplaying. It will still be subjective, just more official. :D
Quote from: ffilz;1122026Yea, you could have an objective measure there as long as you took care with the parameters of the measurement.
Yeah, it's pretty easy to see when, say, swapping between different editions of D&D. My old 1ed group often had 5 fights in a three hour session. My 5ed group often has 1-2.
QuoteCan you play without knowing the rules is going to end up being subjective I think just because it's going to rely on a subjective measure of if the player who doesn't know the rules is contributing in a way that makes you still want to play with them.
Yeah it's a bit fuzzy but it's often cut and dry in practice. Even with the most biased fans if you ask them "can people who suck at learning rules play this game" and you'll usually get a straight answer. Although you sometimes get weird edge cases. Like one player I had who was fine with 3.5ed but simply could not wrap his head around fate points in FATE and just ignored the fate point economy and just rolled his skills, which really doesn't work in that game.
Quote from: Skarg;1122047It seems like good inspiration to make up your own rules, because you sure won't find a complete game in those three books. And yet... an large chunk of folks here will say it's a great version of D&D (though what they mean is, with added knowledge and probably various other books added, plus a good GM).
I had the original brown box. I liked it. It made sense to us as high school students who did not have anyone around who had already played the game. But there was a presumption that DMs would create their own settings and, like most wargaming rules for miniatures that were published back then, the rules were terse and expected the reader would already have certain knowledge e.g. rules for an English Civil War game expected you to have some understanding of how pike & shotte warfare had been conducted and that a battle would progress and conclude like an historical battle of that period so a battle in Cavaliers & Roundheads would be different than one in Tractics regardless of what the miniatures used looked like. Similarly, D&D expected you to know something about medieval warfare, arms, and armor and to be familiar with the underlying fantasy fiction and movies that were later codified somewhere in Appendix N.
Quote from: trechriron;1122113The bigger question, the one that will help nail this conversation down nice and tight is... why do we want to come up with objective criteria for evaluating RPGs?
Isn't that obvious? So we can prove that the game we like is objectively the best.
Objective criteria for games is always conditional. Until all participants in the discussion understand that, is it irrelevant. Once they do, it may be useful.
If we agree for sake of argument that for a particular discussion that number of combats that can be done in an hour is something that correlates to a better experience for some subset of defined people, then we can talk about how games measure on that. If we can set a threshold for number of combats in an hour that is acceptable, for sake of argument, then we can say whether a game hits it or not. Of the games that barely hit it, we can discuss what special cases may cause them not to (e.g. number of players, attention level, etc.). For the ones that barely miss it, we can discuss how the game might be changed in order to hit it. Whether those proposed changes are overall an objective "good thing" is, of course, nonsense. To go there is to change the premise of the conversation. Subjectively, it might still be useful and interesting, but to make it objective will require more conditions.
That some people are bad at objective discussion and/or unwilling to engage in it (for whatever reason) says nothing about what objective discussion is possible or useful--only limits the times and places and groups where it may be possible and useful.
Quote from: trechriron;1122113People complain when the page count is low. People complain when the page count is high. People always complain about the price.
You don't sit down and read HERO or PF2e. You take it in bits. You start playing with the basics (I think HERO Basic is all of 98 pages?) then you dig in. The nice part is you can dig in, because there's a lot of meat on them bones.
The bigger question, the one that will help nail this conversation down nice and tight is... why do we want to come up with objective criteria for evaluating RPGs? Is the current method of evaluation not working? For example, there are TONS of free quick starts, primers and even full books available for people to kick the tires on a game...
But ultimately there are just too many differing tastes, with too many RPGs, with too many approaches to nail this down. Unless you plan on electing someone on the forum God Emperor of Tabletop Roleplaying. It will still be subjective, just more official. :D
Do you even lift, bro?
Quote from: tenbones;1122125Do you even lift, bro?
What? I have both Volume I and Volume II in the porcelain throne room for two fisted curls during my regular constitutionals!! I also read them sometimes!!!
Sales and popularity are very subjective from my point of view. I don't respect the taste of the market very much. Most RPGs, and the most popular RPGs, I do not want to play, because they do little that I like, and way too many things I don't like and/or hate. For the things I care about, they are incompetent and mostly useless to me.
Page count is subjective, and depends on what's in those pages, etc.
Combat time is subjective. Some people love some combat systems, and are happy to do hours of fun combat. Some people just want the combat to be over quickly. Some people want no combat, or no combat system.
Narrativism and player-agency are subjective, unless you can hold a categorization that non-trad games start to not be RPGs but something else, which people can't seem to agree on either.
Can the people in this thread all agree any particular game is objectively bad? Do we even want to mention or think about the games that are that bad?
Quote from: Mishihari;11218983) The rules are clearly written and well organized.
Shadowrun 5e's horrible rulebook turbocharged me thinking about whether this actually does have some potential for objective improvement.
I've been wondering for a bit now if it'd be possible to apply ideas out of programming languages and computability to RPG rules and rulebooks as a minimum scaffolding to improve comprehensibility.
Programming languages have to be computable, and can be quite expressive and close to written languge.
Could we build computable RPG rules to machine enforce that they actually come together to make coherent sense?
One of the things I think this could certainly tackle is editing errors where terms/symbols/abbreviations are used before they are defined, or lack any definition at all, for example. As well as generating more useful indexes automatically.
And if it helps (probably not), this is not a new discussion. I wondered why some of the posts seemed familiar until I realized: the thread reminds me of a column about evaluating campaign supplements back in Dragon #126.
(Yes, I was into D&D then. Don't ask how old this makes me.)
From Ken Rolston's initial commentary (before he begins reviewing some aforementioned supplements):
QuoteSo, what's first on your shopping list when you go looking for a campaign supplement?
"Uh ... it's gotta be yuh know... complete.."
Nope. The last thing in the world you want is something complete. Way too much detail. Maybe you know one of those lost souls who can tell you the name, age, and weight of every person in his fantasy city, or the three principle exports of every one of his fantasy nations. What you want is the illusion of completeness. You want to feel like the whole campaign worlds in there. But exhaustive detail is no guarantee of that sense of completeness nor is it much of an indication of how useful the package is going to be. My mildew herds graze contentedly on rich pastures of exhaustively detailed campaign supplements in my deep, damp basement.
"Naw. C'mon, you know what I mean. It's gotta feel big. . ."
Yes, indeed. A quality campaign supplement has a sense of scope and grandeur. But sheer size may be a poor indicator of its epic vision. Unfortunately, the first test we game fans use to test campaign supplements for quality is heft.
"Hey, feel this! Hea-vy, man!"
Another similar and slightly more sophisticated test we make is to check page count.
"Wow! 144 pages, and only $10."
I will not pretend to be above such simplistic evaluations. Poundage and page count still make a good first impression on me, even though experience has shown that, more often than not, the more text you get for your money, the less thought and skill into presenting it effectively.
Any of this look familiar? What am I saying, of COURSE it does!
Honestly, I think the best measure of a rulebook is 'how easy is it to find X rule at any given time?'. Which makes Bren's comment probably the best for what we're looking for.
Quote from: ffilz;1121901Those are still all subjective.
Well, yes and no. Subjective in the sense of there not being a universal standard. But you can quantify values based on criteria to those, and then judge it by those values.
Quote from: RPGPundit;1122335Well, yes and no. Subjective in the sense of there not being a universal standard. But you can quantify values based on criteria to those, and then judge it by those values.
I guess you could make your own measures. They would still be subjective (how do you evaluate anyone's fun but your own), but you could come up with a scoring system for yourself (and your group) that could be reasonably objective. But the choices that make up the scoring system are going to be mostly subjective.
I guess the rules comprehensibility you could come up with a way to submit a rules set to a control group and come up with a set of questions about the rules set to evaluate comprehension and get an objective score, but it may or may not be generally useful (maybe you rate comprehension by a random group of rule X as very important, and the random group has a hard time understanding the rule, but for the target group of the game, that rule is dead obvious).
Similar debates have run through the social sciences for many decades. At one extreme you have concepts like positivism, which values some sort of objective provability such as tests of statistical significance. At the other end you have concepts like post-structuralism, which is big on relativist and subjective evaluation. Positivism is great but only works when you can isolate variables that are both measurable and significant - which is not always the case, and then sometimes leaves room for debate about the significance of the variable. Subjective evaluations are difficult to rigourously eliminate or expose bias in.
You're asking hard questions and expecting simple answers - you're not going to get to have your cake and eat it.
'Fun' is not an easy concept to quantify, although you could probably design a survey that would tell whether an audience enjoyed the game. Getting a meaningful comparison between two games (or two versions of a game) on an 'enjoyability' metric would be a bit harder. Ergo, it would be quite hard to design a survey that accurately measured whether a version of the game had improved, and more importantly whether that improvement would lead to better sales. The best you're likely to get out of this test is a like/didn't like measure that might give you some idea as to whether you are on the right track. In this case I think the hardest part of evaluating people's responses would be finding ways to measure or eliminate bias from the results. Designing effective studies to measure subjective things like that is hard and getting it right is where social scientists actually earn their keep.
Essential vs. non-essential complexity is another concept that has complexities of its own. Information theory can tell us about a minimal space needed to represent information, but relating that to real text of a real game is a much harder achievement and natural language processing hasn't really done an effective job of solving that problem. So, we're back to editorial taste and opinion about the value of the complexity in a mechanic. Then you've got differing tastes for simplicity vs. crunch. There is a lot of room for value judgements about what the purpose is, whether it has been accomplished or accomplished satisfactorily. Tunnels & Trolls and Harnmaster are both fantasy games, although it could be argued they have different purposes and audiences. Then there is the question of 'Whose purpose matters - the author or the players?' Even the attempt to get an objective evaluation out of this question might not be terribly meaningful.
Clearly written and well organised could be tested through comprehension tests of people who have read the rules to some extent. You might also be able to use applied experimental psychology (human factors) techniques to see things like how many times people have to search through the rulebook to find things. Whether these measures really tell the whole story is debatable but they might give you some useful insight. However, setting up a human factors lab in your basement and finding enough guinea pigs for a sample large enough to draw statistical inference from is likely beyond the reach of a typical indie game design shop. WOTC might have the dosh to do it (although the 5e material suggests they didn't), but you probably don't.
Take a look at anything written by Tim Hartford, The Undercover Economist. He manages to make books about research methods interesting, and shows many interesting examples of creative ways social scientists have taken to isolate significant variables in a complex, messy social or sociotechnical system.
This brings us back to what Linus Torvalds calls 'good taste' - skilled, practiced judgement from someone with the expertise to evaluate whether something is any good. Not everybody has it, but it can be learned - there is a saying that goes 'Good judgement comes from experience. Experience comes from bad judgement.' Judgement is not necessarily 100% reliable, but it is probably the best tool you have for evaluating the quality or fitness for purpose of a role playing game. You will be too close to what you've written to evaluate so find some folks whose judgement you trust and ask them what they think.