Infidelity Support Group
Any relationship in which one partner engages willfully in sexual relations with another outside of the partnership is considered to have experienced infidelity. This breach of trust is often traumatizing for the faithful partner as well as the relationship, and support is often needed to heal emotionally and to decide whether or not the relationship should continue after...

It appears as though he does not want to face the affair, and what led to it. IMO, the cheater must face themselves before a relationship can heal. His retail therapy is a band aid. It happens to be a very common way that people cope with their emptiness.
Let's look at this objectively. He simply has unaddressed issues that impact you. Change is difficult. Unless he makes a huge, concerted effort....it will not happen. The first step, would be him admitting his problem. He appears to be a long way from that.
You can either accept it, or detach from it.
I don't want to oversimplify it....but that is really the two choices for you.
You mentioned that you tell him that you should not be together after an argument. He is conditioned to that. He does NOT believe you will do it. If you DIDN'T after an affair....why would you after an argument. THE BEAT GOES ON.
YOU ARE NOT A FOOL. Stop labeling yourself. You are an imperfect human being that has been hurt.
If you CANNOT accept his behaviour, and it is affecting you.....then YOU have to make the change....not him.
I am experiencing the same lack of fortitude to make......and stick with a decision.
Let's NOT beat ourselves up over it.
Peace
Do things to build yourself up. Be around people who care about you. I went into "hermit" mode for awhile because the feelings of anxiety just kept me from getting out. I'm better now and plan on getting myself strong. You and I deserve better than this!
He doesn't want to face up to what he did and it doesn't sound like he has addressed HIS issues at all. Do not let his issues become yours.
You just know when its time to go. You deserve to be treated well.
That't not failure. Not in my book.
He's not watching out for your heart with comments like that. When you hurt, he decides its appropriate to stick the knife in a little deeper?
L~