Physical & Emotional Abuse Support Group
Abuse is a general term for the treatment of someone that causes some kind of harm (to the abused person, to the abusers themselves, or to someone else) that is unlawful or wrongful. No one deserves abuse, period. Abuse can be emotional, physical, or sexual.
If he wants to see the child, I'd make him go to court for it. The state won't automatically divide custody if there is DV involved. If it goes to court, then you can start building your case as to why your son should not be around this person alone, if at all.
As for the specifics, don't let him get away with changing schedules. It's ALL about control and if you give an inch they take a mile. If you let him have this one Tuesday because he wants to go to the zoo (zoos not being open on weekends ... *rolls eyes*), that means any day and any time are fair game and you no longer have any right to say no. My ex used to do this same stuff - want to change plans by making out he had this fabulous idea that the kid would really love. It's designed to make you look/feel bad for not letting your son go to the zoo.
Stand your ground or you'll find you don't have any.
What a coward to hurt a really small young boy to make him feel like he has control. Sick bastard. Take him to court!
I just wanted to add that one of the things that motivated me was thinking about how my ex would react once my boy got mouthy, and let's face it, they all get a bit mouthy when they are learning boundaries. I knew that my ex was going to be one of those "Disneyland dads" - no proper discipline or routines so that he can be the "fun" one. But that would lead to greater boundary pushing by around now (my boy is nearly 4). And if he pushed the wrong boundary ... the thought terrified me, since it would happen without me around to protect my boy from his father's anger.
Fight for your son. And trust your own gut about what's best for him.