Please don't be scared, I think your doc is doing the right thing though. The reason your nipple has become inverted needs to be investigated and if the mammogram is clear there's every chance you can resume it I'm sure
Always better to be safe than sorry
Wishing you good luck x