The weakest/strongest classes for Insane are the same as the weakest/strongest classes for Normal. So probably alchemist or maybe bulwark.
Necromancer and Arcane Blade are two of the strongest classes in the game. Necromancer has great defense in the form of Sacrifice, and Arcane Blade has Aegis, Stone Wall, and excellent damage. When people say Arcane Blade's early game is "bad," they're usually comparing it to Archmage. Everything that isn't Archmage is bad compared to Archmage. Their early game is still good compared to classes that aren't S tier.
As for why X might be the worst class, let's go through it individually.
I think we can safely dismiss Berserker, Archer, Arcane Blade, Brawler, Shadowblade, Marauder, Archmage, Necromancer, Sun Paladin, Reaver, Corruptor, Demonologist, Stone Warden, Paradox Mage, Temporal Warden, Mindslayer, Possessor, Adventurer, Sawbutcher, and Psyshot as "not the worst in the game." Everyone knows these classes have something that puts them miles ahead of Bulwark, even if it's a single talent like Bone Shield.
That leaves Bulwark, Rogue, Skirmisher, Alchemist, Doombringer, Summoner, Wyrmic, Oozemancer, Anorithil, Cursed, Doomed, Solipsist, and Gunslinger.
I will eliminate Gunslinger because although its defense is awful, it has higher-than-god-tier damage output due to Awesome Toss. It doesn't make sense to peg that as the worst in the game unless every other class were like, literally invincible, which isn't the case.
Cursed and Doomed have Curse of Nightmares and Cursed Sentry, so those are out too (and Doomed isn't that bad even without broken curse stuff, but people don't seem to be ready to believe that yet).
Anorithil having Celestial / Light from the start is enough to eliminate it alone IMO, and it also gets to chain-brainlock things with Circle of Sanctity, use Jumpgate, Corona, etc.
I will also, without any attempt at exhaustive proof, submit that Rogue, Skirmisher, Summoner, and Solipsist can be safely eliminated. Rogue has quite a bit of good stuff between Mobility, Artifice, Expose Weakness, and Shadow Dance, giving it superb damage and at least passable defense. Traps are also still good if you actually have the patience to use them.
Skirmisher is obviously almost strictly worse than Archer but it still has Mobility and great damage from a long range.
Summoner is really good at not taking damage, because summons, and war dogs and minotaurs have no problem putting out a ton of damage. Maybe if the AI's target selection didn't outsmart itself Summoner would have more problems, but it doesn't look like that's going to change any time soon.
Solipsist has three different ways to Sleep things and Inner Demons, plus Dream Smith is actually very good and so is Maelstrom.
This leaves Bulwark, Alchemist, Doombringer, Wyrmic, and Oozemancer.
Bulwark's damage is fine, maybe it's even a bit above median. It's melee and it has Shield Offense and GWF. The problem is everything else - its only claims to defensive advantages are some extra max HP (most of which is baked into a talent that pins you as a melee character and takes away current HP when deactivated), stun immunity from Shield Wall, a little extra armor/defense from Shield Wall, Conditioning, and True Grit, a talent that works so poorly that I almost decided against mentioning it at all. And other classes can get Vitality from escorts. Its only mobility talent is Step Up, which is good but not exceptional, and its only status removal talent is Unflinching Resolve.
So Bulwark is among the worst in the game at everything except damage. Gunslinger probably has worse defense but it has the aforementioned super-god-tier damage, which Bulwark does not.
Alchemist has unquestionably some of the worst damage in the game, and it doesn't have status cures or good mobility. Its basic damage is still usable for most of the game and talent_on_spell mostly solves its damage problems later in the game (it won't on Madness but we're not talking about Madness) so it's not unwinnable or anything.
You can make an argument that Alchemist is better than Bulwark because you can use the golem much like you used old Ambuscade: teleport it, let it fight things, Supercharge Golem to resurrect it when it dies. Teleporting allies long distances is very hard for the other classes in contention for worst in the game, but Alchemist can do it easily with a teleport rune. In combination with a properly dug tunnel (or even better, an island), this is actually pretty good, and it's noncommittal as you only need to control the golem for long enough to teleport it. However there are some levels this just doesn't help on. Alchemist is also infamously bad in the final fight, but if you have Life Drinker or another good talent_on_spell I assume it's fine.
Ice Core lets you reach 100% crit shrug really early (thanks, rares with 15% crit shrug) so you can also argue that Alchemist is defensively better than Bulwark lol.
I think Bulwark is probably still a bit better overall though.
Doombringer's defense is at least as good as Bulwark's, its offense is obviously way better, it has Wraithform and a ridiculously good status cure. Not much else to say.
I don't think Wyrmic is a serious contender for worst in the game because if nothing else, Ice Wall and Lightning Speed carry it. It has a few other great talents like Dissolve and Burrow (though you can get Burrow from mindstars...) but really Ice Wall alone is just too good.
Oozemancer's damage is perfectly good once you understand how to use mucus oozes and its damage reduction is fine too. It has an excellent teleport and status immunities. The only reason you might regard it as the worst in the game is if you regard Antimagic even more poorly than I do: it's the only class in the game that can't use Stone Wall or Aetherwalk and it doesn't have anything resembling a replacement for Stone Wall either. It also has a lot of difficulty exploiting Arcane Amplification Drone. However, seeing as you're not guaranteed to find Stone Wall items, you're not guaranteed to find Aetherwalk and even if you do it won't be early, you don't get AAD until level 30 and it's not that amazing without Garkul's Revenge (or Touch of Death), I think this argument is implausible.
Note that even the worst classes can win Insane comfortably. There are several insane bulwark and alchemist wins in the vault.