Both in the crunch (rules) and fluff (setting).

I'm someone who likes to have a lot of rules dedicated to what I can and cannot do in ttrpgs. For me, a badly designed magic system would be both vague and restrictive. As in "You can only cast this spell once per encounter, but first you need to fiddle with your clock and dice pool values to reach a value that your DM arbitarily set, and then you can cast a spell, whose effects and damage, if there are any, are explained in a few words that are only more confusing".

I am pretty open minded about magic systems, but I remember the 2E psionics approach really bothering me (I suppose not technically magic but I would still regard it as a magic system)

When it's too open-ended that it leads to endless arguments. Granted that's a lot of spells throughout different systems, which is a good reminder about clarity of wording and defining limits. But I am also thinking about White Wolf's Mage: the Ascension... aaaand now I am getting flashbacks sitting through discussions about what a Mage can do. /sadface

Did you ever play Mage: The Awakening?
I only played that and it . . . seemed pretty good. Space to create your own, but enough there that you didn't have to and the examples provided good context.

Mind you I didn't play this one as much as I wanted to at the time.

So, this is one I've put a lot of thought into.  I like GURPS Magic, AD&D, Rolemaster, Dragon Quest.  I'm not fond of Runequest but I do like the Worlds of Wonder / Magic World magic system.

Personally, if the magic is really tied to a setting the problem becomes that it's too restrictive.  Did they cast levitate in Lord of the Rings?  Well then this magic system doesn't fit because tightly defined settings become too exclusionary.

I guess it's really why I think licensed setting make bad rpgs as well.  Rpgs should be about exploration and creation not being locked into a cell.

Lack of consistent limits and specifications. As per Brandon Sanderson, it's often more important to a story -- or consistent game environment -- what your magicians can't do than what they can.

Almost as bad is a magic system that doesn't make useful options available for whatever the PCs are supposed to do. If the game is about killing monsters and taking their stuff, magic which doesn't make it easier to kill things and find stuff isn't going to see much use.

WotC's D&D-ish game is the worst. It lacks limitation so the casters easily dominate the game. Pure trash.

I'm not a fan of DCC but the magic system is dangerous for the casters and their teammates. That's a good thing. Even something like TRoS where using magic causes casters to age faster is good.

The late E.G. Gygax called WotC's rpg a superhero game and he was correct.

I agree with most of what's been said above about bad magic in crunch. If you want to see a magic system trip at every single hurdle, check out Modiphius' "Conan: Adventures In An Age Undreamed Of". Incomprehensibly vague spell descriptions? Check. Multiple steps to resolve every spell? Check. Extremely situational spells that will probably never be used? Check. Having to negotiate the affect of every spell with the GM? Check and check!

I'm not sure there is such a thing as bad magic system in fluff, fluff being almost entirely a matter of taste. It really only becomes an issue when the fluff and the crunch contradict each other. If magic is supposed to be rare and dangerous, you can't make it both highly useful and free (as it is in the majority of RPGs). If magic is supposed be channeling arcane energy out of the environment or summoning up otherworldly creatures, then neither memorizing spells or an MP system makes much sense. Etc etc.

It is difficult to tell what would be bad for a particular setting without knowing anything about the setting. What magic is capable of, should be a part of the setting and somewhat apparent to anyone exploring that setting unless magic is some sort of recent development. If magic is super powerful then the the power structure should reflect that. If magic is lower powered and subtle, the same applies.

The types I personally disdain are the ones that are more or less purchased feats or powers. those to be are uninspired and not even worth my time, they don't fit into the fiction, rarely are balanced and don't feel very magical to me.

So this is most GURPS magic, or anything produced by Green Ronin.

Magic systems I think are the peak of design would be GLOGs magic dice system, because the idea of taking Vancian magic one step further with the idea that spells are non-fungible, that my version of fireball and your version of fireball are discrete entities and only happen to look alike.

LIkewise while I wish it was more concise, something between Ars magic 2nd and Ars magic 5th edition magic is just wonderful, I'm trying to adapt that system in a more concise way into an OSR system, but a semi-open system where GM discretion, spell precedence and the fiction of the setting all helping you triangulate on what a spell should do at what cost is kind of awesome.

Magic without limits or consequences are the ones I dislike the most.

Both in setting and systems.

What Exploderwizard said, plus the Brandon Sanderson approach is half right:  Yes limits are important, but also important is how far magic takes over science/tech versus how much it is supernatural/mythic/fairy tale. 

The more logical and repetitive magic is, the easier it is to fit into a setting and have limits.  It also becomes less "magical" in the process.  This is also true if the magic is common.  So when you have wizards hiding under every rock and popping out to blast things, it's easy for it to turn into space troopers hiding under every rock popping out to use a blaster.  That's fine if that's what your after. It's horrible if it's not.

Which brings us back to Exploderwizard's point, hard.  The setting must be the prime driver. Rules are downstream from that.

Personally, I think rules for magic or magic "systems" in general are overrated in prose fiction. I don't think there's a single fantasy franchise I like which has a "hard" magic system. I prefer magic which is ill-understood, even by the characters in the fiction. I was going to say that I disagree with Sanderson as regards non-interactive fiction, but I went and read this article (https://www.brandonsanderson.com/sandersons-first-law/), and I think he has it mostly right. This quote in particular:

Quote"The really good writers of soft magic systems very, very rarely use their magic to solve problems in their books. Magic creates problems, then people solve those problems on their own without much magic."

And that is true of almost every piece of non-interactive fantasy fiction that I hold in high regard. The only major exception would be Michael Moorcock, who is periodically guilty of letting Elric pull a big spell out of his ass to solve problems. But in that case you could argue that the monster of the moment is never really the threat to Elric, so much as the overarching threat of Arioch and Stormbringer.

In games though, the soft magic approach is borderline unworkable. Games aren't much fun if the player can't make informed decisions, and to do that they need to know how the tools at their disposal work. You have to make magic almost entirely unavailable to the players, which would be the right approach in something like a Conan or Lovecraftian game, but not much use elsewhere.

RPG magic systems are like cooking. Too much salt ruins a dish, but not enough means it will taste bland. Too much chili powder will make it inedible for most people. Even if your take on it is the best on the planet, nobody wants to eat bat soup. Topping a salad with dish soap instead of vinegar is probably a terrible idea. However, soap chemistry is an important part of deep frying; the salt in the food reacts with the fats in the oils to produce soap, and the chemical contact these soaps enable between food and oil is a key component to the perfectly even browning unique to deep fried food.) 

My point is there are many, many ways RPG magic systems can fail, not just one or two, and the more you know about them the more nuanced truths flip what you think you know on your head. Most RPG magic systems have some faults, too, so just having them isn't necessarily a deal-breaker either. No, they fail when a flaw or combination of flaws goes supercritical and starts interfering with the other components of the game. Most often, magic starts hogging gameplay time or player spotlight time. Other flaws tend to not actually break magic systems so much as dilute your enjoyment of them.