I gave a purple gem to the Relic shop owner because I wrongly thought you could sell gems to him. His desk was so offended by my offer that it erased itself out of the space time continuum, and now the poor fella is without a desk. It's physically gone, I can walk through the area where the desk once was so it isn't just invisible.
Leaving and coming back doesn't change anything either
Give purple gem to relic shop owner