I recently started a new RPG design project and could use some feedback and advice.

The basic premise of the game is grim-and-gritty medieval adventure. Think "Warhammer" but without the "D&D" elements: no demihumans, no orcs, no PC spellcasters. There is no explicit setting, but thematic elements (war, disaster, plague, chaos) are implied. I am trying to designing the system and the implied setting in as modular a fashion as possible, allowing campaigns to run the gamut from historical adventures with no magic whatsoever to Lovecraftian horror.

Because magic and the supernatural are out of the hands of the PCs, and in fact might be totally alien to them at the beginning of the campaign, I would like to include some optional rules for fear and insanity. I have been reading a lot about player agency lately, and it has caused me to modify the way I run and develop games. As such, I would like to design the fear and insanity rules in such a way that players still keep at least some degree of control over their players actions in these kinds of situations. For example, I am trying to avoid situations like this:

GM: Make a fear (insanity) check.

Player: I failed.

GM: You flee in terror (go insane).

Player: What? I don't want to do that!

GM: Sorry, those are the rules.

I could, in theory, utilize a system of penalties to actions. For example, if the character fails a fear check, they might suffer negative modifiers to certain tasks, which could be eliminated by fleeing. Insanity might reduce a character's mental or social abilities, but it wouldn't remove control of the character form the player.

How would you go about designing these mechanics? What fear/insanity rules have you used at the table? What worked? What didn't work?

As an additional note, I am designing the system to reflect a dangerous world. Characters are fairly ordinary mortals, even if they are warriors and adventurers. Their abilities can improve over time, but combat and other hazards can always present the risk of death. Characters must learn to be selective about the actions they take, or else they will most likely die. It might be that this aspect  of the design will be enough to convey the "fear" that I want to be present in my game, at least from the characters' points of view.

From my real life experience, there have been times I've been terrified and frozen. In martial arts, there have been times I've been afraid of the person I was sparing, but not backed down, but become inhibited and ineffective. I've never had a wizard cast magical fear on me, but I imagine that would be worse.

So I think you could have a couple types of fear: natural terror, unnerved, and magical terror.

Natural Terror is when you freeze because something you didn't expect and can't instantly deal with appears, like a big snarling dog standing across the street that you notice way to close. For this, I'd give a save vs. paralysis every round.

Then there is unnerved, which is when you are able to act normally but at a lower level of performance. For this I'd give a one time save, failure imparting a penalty on rolls.

For magical fear, save or run.

This is effectively how Pathfinder does it with their different levels of fear: shaken, frightened, and so on.

Holy crap, Cranewings. That sounds perfect! Pathfinder, you say? The core system I am using for the game is put together from bits of FATE and d20, so I am already using the Open Game License. Do you know where I could find these rules? I am curious to see if it is something I could just import into my game under the OGL.

EDIT: Nevermind, I found the conditions in the System Reference documents for both Pathfinder and d20. I think I will give these a shot and "borrow" your suggestions on their implementation. Thanks for the tips!

No problem sir.