This is one of those things that might just better if you have it removed. It doesn't matter if its not cancerous, dangerous, etc. It seems to be damaging your quality of life and your emotional state.

I'd just have it removed. You can do elective mole removal. And if he won't do it, find someone that will.

I hope it works out for you.