ToME: the Tales of Maj'Eyal

IIRC in Tome 2 it affected how much a skill increased for each skill point spent. But in Tome 4 each skill point increases a skill by 1, to a max of 5. So what does mastery do?

It's a multiplier on the talent effects.
it does not affect all talents, but most.
So if you have (example) a spell doing 100 damage at level 5 with a mastery of 1 it might do 130 at the same level with a mastery of 1.3

And weapon mastery?
Do they increase the damage done by the specific weapon? Or the spell damage?
In fact, with, for instance, a Two-Handed mastery amulet, I see no difference either in the spell effects in the talent window ('m') or in the damage of character sheet ('C')...

Well what it really does is increasing the actual talent level used by the code, so at 1.3 a level is 5 is 6.5.
This should affect all talents (well most of them)