ToME: the Tales of Maj'Eyal

If I have armor that says "x damage on hit". Does this cause damage to me or to the enemy who hit me?

To the hitter. So, usually, to the enemy - but guess what happens if you try to hit yourself?

There are two very similarly named but opposite effects. Neither is a penalty.

Damage on hit applies damage when the wearer hits something.
Damage when hit does retaliation damage when something hits the wearer.

The type of item the effect is coming from doesn't influence its operation at all and while I can't remember seeing the retaliation version on weapons I have seen both on armor.

So the answer to your question is "No," because it doesn't do anything at all when an enemy hits you. It's for boosting your hits.