Thoughts of the last two days

Yesterday husband and I had a big blow up because my son is upset and doesn't want to see him this coming weekend, and some how i turned into the bad guy. He blames me for our relationship falling apart and that that for the last two years we have slowly been growing apart. I went back through conversations last night cause i wanted to see if i could pinpoint a time that things started to fall apart for my own benefit and it was about last September we had a big fight and the same conversation came up about him not being happy and there being a high level of stress and that he was sick of trying to make it work. There was a high level of stress i had lost my job a few months before and he was providing for us so I understood the angry and frustration and we got past it cause a few weeks later i had found a job. He still had the stress from lack of money and his brother in law was living with us and causing big issues between us, which didn't help. It didn't matter how many times I asked for us to move away from his brother he kept staying and dealing with it. His brother got so angry one day said if he had his way he would kick me out of my own home because he dislikes me that much. My husband came home and blew up and finally took my side however shortly there after he apologized to him for blowing up and they were fine but the damage in my relationship to the brother had been done and we weren't talking. I finally got my husband to agree to move we were going to buy our own home and we (i thought were happy about this). It was tough cause not many houses in our price range and area we wanted to stay in. I just wanted it to be us and no more of his family living with us. We were so much better when it was just us and we were happier. But he decided that his family should stay with us for extended amounts of time and then his brother to live with us. Anyways he took the money we had for the down payment and paid off my car and then we kept just talking about renting for a while and that he wanted to move but not now. Then it all fell apart again and we fought and this time there would be no recovery.
 
The moment I decided to stay a few nights away from the relationship he ended it. He puts it on me that if I hadn't left then we might still be together. But how is that better for me. It was like living with a roommate that slept in the same bed. I would say I love you and get a response but it wasn't with meaning and he wouldn't say it first. He stopped hugging and kissing me, he pulled away to the point that he was done he had said that he was ready to get the paper work and sign away our marriage and be done. But he hadn't done any of that because at the time I was fighting with my parents and I have no other family in town so I couldn't go to them and thats why he didn't tell me to leave earlier in the month.  I told him to make a choice you actually want to fix this or you want me out and he said go and he would take the next few days to think. So i went thinking it would give him some time and two days later he wants out and he is done and just because I left and didn't stay when he said go its my fault again. 
 
I am so confused. Today he is talking to me like there has been no damage and that we are just fine, and that its just another day. I am trying to move on and get my own place so that my son and I aren't cramped in with my parents and its when he talks to me about random things that makes me think he is just taking the easy way out. I can't and wouldn't throw away 4 years for nothing and every couple fights and has their bad moments but its that, that defines the relationship, but when you just give up what does that say about you. Today makes it hard to move on when I just keep hoping(deep inside) that he will come back, but i am sure later today he will make his point know that he is done. I am done to but i can't stop that part of me who hopes that maybe it will work itself out before we sign our marriage away that maybe he will come to his senses and choose me.. Dang it want the happy ending not the sad miserable one laid out in front of me now.