I had a crazy idea, namely that a massive emphasis on playtesting could save TRPGs.

Obviously professional soldiers write AARs regarding their training sims, and computer game developers have various documents about software testing.

However, in TRPGs, it is practically impossible to find documents that deal with playtesting.  Most companies don't playtest much, don't document their playtesting, and certainly don't release playtest report documents outside the company.

To open the discussion, assume D&D is the system at hand. Ideally I would like to see playtesting on three levels: tactical,operational, and strategic.

The tactical level would be one encounter.  "Here test group 49-A entered with one rogue and one cleric.  The cleric tanked while the rogue disabled the slow-drowning-in-treacle trap.  They both had high dexterity so the to-hit disadvantage from wading in treacle didn't slow them down and they killed the monsters in two rounds, suffering only 5% total hit point loss, bringing them down to 80% of full health.  They chose to exit the room by the secret door, thus avoiding the orcs..."

The operational level would be one series of modules and might include one or two level-ups.  "In the second module, group 67-C chose to negotiate with the evil god without becoming evil.  Thus their cleric became useless and they rapidly ran out of healing potions.  They escaped without loss of life, but the playtesters were so demoralized that three out of four dropped out of the playtest.  The fourth wanted to switch classes."

The strategic level would be the entire campaign.  "Over the course of six months, group 14-D leveled their characters from level 1 to level 30.  Their emphasis on building divination skills reflected their dissatisfaction with the DM's limited clues. While weaker in damage-dealing than most groups, this group tended to breeze past modules with minimal interaction and maximum circumvention of foes, treating the game like a puzzle instead of a battle.  The DM's biggest complaint was that this group demanded XP rewards without fighting or even risk of death.  The group proved very capable of out-strategizing the DM, killing off his metaplot characters one after the other until the DM stopped trying to push the metaplot and allowed the players to dominate the campaign story."

On the biggest scale, this kind of testing would require multiple groups for each DM.  However, considering the massive existing player-base of D&D, I think a large number of tested is entirely realistic.
 
More difficult would be maintaining the motivation of playtesters.  Software companies pay testers very little and demand very little skill.  TRPGs require more cooperative feedback, and the testers themselves would probably need to at least verbalize their satisfaction levels.

Further, standardized testing would be easier with a group of a standardized size using pregenerated characters.  Conversely a lot of gamers insist on making their own characters with their own individual powers and a group of nonstandard size.  Thus playtesters who adhered to a company plan of pregen characters would obviously be doing work and obviously would deserve pay.  

Now, getting a reasonably responsible TRPG company to do playtesting is the easier part of this.  Getting the documents out to the customers is the hard part.  Many companies are willing to do some testing and admit to their CEOs that the designer was an idiot and that the playtesters complained about obvious idiocies.  But those same companies are not willing to publish a book about their playtesting and design process, even though that book might make it easier for DMs to avoid the designers' mistakes.

In my mad little dreams I would like to see a module where 50% of the DM's material was tactical advice based on play-testing.  I know that's an insane dream.

You know what might help here? Online TRPG games, via AIM. In the process of playing, players produce their playtest document (a chat log). Yeah, there would still need to be feedback at the end of it, but even if that turned out minimal, you'd still have a log of the session and a pretty good idea of all that happened.

Quote from: beejazz;298755You know what might help here? Online TRPG games, via AIM. In the process of playing, players produce their playtest document (a chat log). Yeah, there would still need to be feedback at the end of it, but even if that turned out minimal, you'd still have a log of the session and a pretty good idea of all that happened.

Further, a text record would be much easier to read through quickly than (e.g.) a videotape.

The process has to be automated, if you have your standard "player" types doing the tests.  

You see players are in it for the fun. It has been my painful experience that they write awful responses that are either useless or non existant. (Verbal interviews are better). Unless you are paying them to play, they tend to see this "extra work" as, well, work they are not getting paid for.  They don't do it.

To be honest, most playtest groups tend to get "group think".  They are playtesters in a given game group because they match the play style.  This becomes Tri-Tac/ GURPs/ Hero/ Palladium dedicated groups.  They play the game and kind of enjoy it. Yet, when you take the game out into the world, despite the playtesting you have done, people outside that game group despise the specific game.