Panic attack and frustration

Alright... So today I got a big scare... My ex I recently broke up with showed up at my work today and I had a panic attack.... It was bad. My heart was racing, my body was shaking, I couldn't breathe... He wasn't there long, but long enough I about lost it. 
Then I come home from work to fully establish plans for tomorrow as a birthday dinner for me and my brother. And part of the plan was to see if this newer guy I was sorta seeing would be interested in joining us cuz we had an extra spot reservated just in case. He has yet to meet my family, he's too shy is what he tells me... From what I told my folks about him they think all he ever wanted from me was sex... Which is starting to be more clear to me as each time I offer he join us/wanting to hang out away from his place he turns me down... Pisses me off... I wanna try for something but he isn't interested in that. 
He told me from the start that he isn't wanting a relationship because he is worried he'd end up hurt in the end. Yet we have gone and had sex a few times at his place. I just don't get it. I'm crazy about him, but I know its not worth it. Ugh! Guys can be such butts!