Magic is a powerful tool. However, that does not make it bereft of flaws and just like all other tools the path of magic will leave an individual lacking in certain aspects. There are many ways in which one can power their magic, however, regardless of magic type these magics find themselves always tending to one result: Magic makes one physically weaker. The reason behind this however is different for each magic type:
Arcane: Individuals who follow the arcane branch find themselves attacked by two barriers to physical strength. The first being that they spend much of their time studying the arcane, however that alone is not the root cause of why they are incapable of being particularly strong. The true root comes from the void itself. The power of the void coursing through one's body is an unnatural thing, particularly when one's opening to the void is so small. It is therefore the void acts like a toxin, sapping the body's strength and rendering it impossible to build more than a small amount of muscle. Such is the curse of the mage.
Necromancy: Individuals who necromance are weak for a similar reason that arcane mages are weak. Their power source is something unnatural to deal with. The very shifting of life force weakens their bodies and just like the arcane mage they can not build such muscle.
Soul Puppetry: Curses to a degree act on the user with their corrupting power. Making them weaker as their body is constantly under corrupting stress.
Contract Magic: To properly write the contracts, the user would need to spend extensive time reading and understanding the words and their meaning, as well as properly writing and carving perfect symbols. This would turn the user more into a scribe than a fighter, but the magic itself does not harm or weaken the physic. Some fighting capability is retained, but they would never be able to defeat a warrior in a one on one fight.
Holy Magic Users: One would not think that a holy mage is subject to weakening, however such is unfortunately the case. Just like the power of the void, the power of an aengul is corrupting and withering.
Frost Witches: Cappy's Explanation
Fi'hiiran'seth: You guys have 2 void connections. Think of the arcane troubles and then multiply them by two.
Druidism: Druidism expresses its weakness in a different way. Due to the connection to nature druids are simply more susceptible to being 'lost in creation' when in such a close-proximity-fight. The aspects grant the druids health, somewhat negating the weakness the power gives to other divine magic users.
Kharajyr Magic: Same fault as Holy Magic Users.