Using the latest Git, there is a bug with
Tirakai's Maul gem imbuing. When replacing the gem it returns not a single one, but a stack of gems, equal to the remaining gems of this kind (i.e. if I have 5 garnets and imbue one, I'm left with a stack of 4; when replacing the garnet I will have two stacks of 4 garnets). This is the half of it, though. After the action is complete, both identical stacks will be deleted and I'm ending with a single gem of this kind in the inventory. That will teach me not to meddle with arcane crystal things

.
I'm playing an alchemist and the golem is wielding the maul, so maybe that also has something to do with the weird behaviour. I've done the swapping first with the main character and saw that I was missing some gems, so I've done it with the golem and can confirm the bug. I have even given the extra stack back to the alchie and it did the same thing in her inventory (deleting all and replacing them with the gem that was just removed).