Physical & Emotional Abuse Support Group
Abuse is a general term for the treatment of someone that causes some kind of harm (to the abused person, to the abusers themselves, or to someone else) that is unlawful or wrongful. No one deserves abuse, period. Abuse can be emotional, physical, or sexual.

My ex was physically abusive, but he now lives a half a world away - literally. He can't get to me, he doesn't even know where I live. And yet, when he sent me an on-line message recently, it sent me into a spin. It's our natural defence mechanisms. We know these men are dangerous to us and so we are naturally going to be afraid of them, even if the danger is "only" to our psyches.
Go easy on yourself, hold off on the name calling ("silly" and "weak"), the minimisation (there is no such thing as "benign" abuse), and also less of the "should" when it comes to emotions. You "should" feel whatever you feel - there is no right or wrong when it comes to emotions, they just are. If you are afraid, then accept that, and know that it's your psyche reminding you that this man is dangerous to you, even if he's not physically dangerous.