Quote from: ffilz;1121901Those are still all subjective.

Well, yes and no. Subjective in the sense of there not being a universal standard. But you can quantify values based on criteria to those, and then judge it by those values.

Quote from: RPGPundit;1122335Well, yes and no. Subjective in the sense of there not being a universal standard. But you can quantify values based on criteria to those, and then judge it by those values.

I guess you could make your own measures. They would still be subjective (how do you evaluate anyone's fun but your own), but you could come up with a scoring system for yourself (and your group) that could be reasonably objective. But the choices that make up the scoring system are going to be mostly subjective.

I guess the rules comprehensibility you could come up with a way to submit a rules set to a control group and come up with a set of questions about the rules set to evaluate comprehension and get an objective score, but it may or may not be generally useful (maybe you rate comprehension by a random group of rule X as very important, and the random group has a hard time understanding the rule, but for the target group of the game, that rule is dead obvious).

Similar debates have run through the social sciences for many decades.  At one extreme you have concepts like positivism, which values some sort of objective provability such as tests of statistical significance.  At the other end you have concepts like post-structuralism, which is big on relativist and subjective evaluation.  Positivism is great but only works when you can isolate variables that are both measurable and significant - which is not always the case, and then sometimes leaves room for debate about the significance of the variable.  Subjective evaluations are difficult to rigourously eliminate or expose bias in.  

You're asking hard questions and expecting simple answers - you're not going to get to have your cake and eat it.

'Fun' is not an easy concept to quantify, although you could probably design a survey that would tell whether an audience enjoyed the game.  Getting a meaningful comparison between two games (or two versions of a game) on an 'enjoyability' metric would be a bit harder.  Ergo, it would be quite hard to design a survey that accurately measured whether a version of the game had improved, and more importantly whether that improvement would lead to better sales.  The best you're likely to get out of this test is a like/didn't like measure that might give you some idea as to whether you are on the right track.  In this case I think the hardest part of evaluating people's responses would be finding ways to measure or eliminate bias from the results.  Designing effective studies to measure subjective things like that is hard and getting it right is where social scientists actually earn their keep.

Essential vs. non-essential complexity is another concept that has complexities of its own.  Information theory can tell us about a minimal space needed to represent information, but relating that to real text of a real game is a much harder achievement and natural language processing hasn't really done an effective job of solving that problem.  So, we're back to editorial taste and opinion about the value of the complexity in a mechanic.  Then you've got differing tastes for simplicity vs. crunch.  There is a lot of room for value judgements about what the purpose is, whether it has been accomplished or accomplished satisfactorily.  Tunnels & Trolls and Harnmaster are both fantasy games, although it could be argued they have different purposes and audiences.  Then there is the question of 'Whose purpose matters - the author or the players?'  Even the attempt to get an objective evaluation out of this question might not be terribly meaningful.

Clearly written and well organised could be tested through comprehension tests of people who have read the rules to some extent.  You might also be able to use applied experimental psychology (human factors) techniques to see things like how many times people have to search through the rulebook to find things.  Whether these measures really tell the whole story is debatable but they might give you some useful insight.  However, setting up a human factors lab in your basement and finding enough guinea pigs for a sample large enough to draw statistical inference from is likely beyond the reach of a typical indie game design shop.  WOTC might have the dosh to do it (although the 5e material suggests they didn't), but you probably don't.

Take a look at anything written by Tim Hartford, The Undercover Economist.  He manages to make books about research methods interesting, and shows many interesting examples of creative ways social scientists have taken to isolate significant variables in a complex, messy social or sociotechnical system.

This brings us back to what Linus Torvalds calls 'good taste' - skilled, practiced judgement from someone with the expertise to evaluate whether something is any good.  Not everybody has it, but it can be learned - there is a saying that goes 'Good judgement comes from experience.  Experience comes from bad judgement.'  Judgement is not necessarily 100% reliable, but it is probably the best tool you have for evaluating the quality or fitness for purpose of a role playing game.  You will be too close to what you've written to evaluate so find some folks whose judgement you trust and ask them what they think.