Physical & Emotional Abuse Support Group
Abuse is a general term for the treatment of someone that causes some kind of harm (to the abused person, to the abusers themselves, or to someone else) that is unlawful or wrongful. No one deserves abuse, period. Abuse can be emotional, physical, or sexual.

When I kicked my ex out, for a while I was also more afraid than I'd been when he was living here. I could have no idea what his mood was, whether he'd been drinking, what he was doing or thinking - and I had no way to work around it and make sure I was safe, no warning. I also used to leave the house when I knew he was likely to kick off, but once I was alone I had no way of knowing when he might kick off.
If he turns up at your house again, call the cops. Make clear to him that that is what you will do. Fortunately, my ex didn't actually turn up, but he threatened to - so I had my lawyer write to his, saying that I had contacted my local police and his parole officer, that if he turned up I would be dialling emergency. Because I had got his lawyer, parole officer and the police involved already, he knew I was serious about calling - and that he'd probably go back to jail if he turned up.
I also jumped every time the dogs moved, or I heard a noise or ... anything. I slept with my phone by my side. I triple checked that the house was locked before sleeping (I never used to lock my house at all, even when I went away for a few days, much less when I was inside it). But slowly, the fear subsided and although I still make sure the doors are locked when I'm at home, I no longer jump. He's been gone more than 2 and a half years now. It did take months to calm down.
You are not crazy.
It's for these reason's that establishing safety is such an important first step in recovering from abuse. You need a secure home and whatever legal protection you can get. I hope it all works out for you.