Physical & Emotional Abuse Support Group
Abuse is a general term for the treatment of someone that causes some kind of harm (to the abused person, to the abusers themselves, or to someone else) that is unlawful or wrongful. No one deserves abuse, period. Abuse can be emotional, physical, or sexual.

I know it's not that simple - it's not like I left my abuser at the first sign of abuse. Or the first time he hit me. Or even the first time he dumped me. But I wish I had.
To take your specifics, though, you say he only physically abused you once, and that he was drunk. Has he had a drink since? What has he done (as opposed to said) to ensure he will never ever do it again? I don't believe alcohol is any kind of an excuse, and I don't think that an abusive person becomes non-abusive by not drinking. I do think, though, that a sure sign that it is an abusive person rather than someone who acted once under the influence and entirely out of character, is how they respond to what they have done. That is, what does the person actually DO about what happened? At the very least, he should have taken real steps to ensure he never drinks again.
On top of that, you say your self esteem has plummeted, so I would guess that the one instance of physical abuse is not the extent of the abuse, and that there is more regular emotional and/or verbal abuse going on as well. Probably also carried out (usually) while under the influence of alcohol, and followed by apologies afterwards. Again, this indicates an abusive personality, rather than someone who just acted out once.
If my guesses are completely misplaced, I apologise. If they are not, then please read up on emotional abuse (there is loads of very good info on this site and elsewhere around the net), and think long and hard about what he is really doing for you, and what he is doing to you. And think about whether this relationship is healthy for YOU.
Sorry if I appear harsh, that's just my take on it. Please take care of yourself, and do what's best for you. Whether you believe it or not, you ARE worth that much and more.
Feel free to post here as often as you want. You will find people who understand, and who don't get sick of hearing about it - because we were or are there ourselves.