Here's how it works: Like you said, when you give a golem a gem, it takes the whole stack. If you then attempt to change the stack of gems to something else, it will use up the new gem (or stack) you give it and it decreases the stack of gems in the golem by one. You have to keep feeding it gems to ...