Why is he punishing me ??

When my little girl was born 10 weeks early, I didn't know where we would be a little over a year later.  We had no idea how things would go, and I honestly thought her father was going to leave me.  He didn't want children, and we had one.  He didn't leave.  He stayed, and things had gotten so much better.  But now my baby is 14 months old, and it seems like we are slowly drifting apart.  He doesn't seem interested in me most of the time, and doesn't much want to do anything with our daughter.  He basically told me one day that it's my own fault that I am stuck home with the baby everyday because I chose to bring her home.  Then last night he said something about money, and I mentioned how I pay for everything for the house and our girl, and he told me that was all my responsibility because he wasn't claiming her.  It's not fair, it seems like he is blaming me for his life not being what he wanted.  He gets to go out when ever he wants, and do what he wants, spends his money on what he wants.  I spend all of my money on everyone else, do everything at the house, and take care of the baby.  Yet I can't get five minutes to myself.  Why don't I deserve a chance to go see a friend? Why can't I have a break?  It's like he's punishing me for loving our baby too much to let her go.  How is this fair?