Physical & Emotional Abuse Support Group
Abuse is a general term for the treatment of someone that causes some kind of harm (to the abused person, to the abusers themselves, or to someone else) that is unlawful or wrongful. No one deserves abuse, period. Abuse can be emotional, physical, or sexual.

I'd tell him to back the f off and never contact you again.
It's like he's already trying to control you by expecting you to be okay with what he did and just b/c he said that he'd never hurt you means permission for him to invade your space like that.
I'd start documenting this stuff. It'll escalate...