Anxiety Support Group
Anxiety is a physical condition marked by intense and persistent feelings of distress, fear, angst or dread. General anxiety caused by routine day-to-day stresses usually passes quickly and is experienced by almost everyone at one time or another. However, such feelings that linger over time and are very difficult to cope with, and which lack a clear cause, may indicate...

There are always going to be people judging you and you have to be confident in the decisions you make. If I feel like I did the right thing then I don't worry about what others say.
I don't feel you are selfish. I believe you are taking care of yourself which is the right thing to do. It sounds to me, based on what you wrote, that your family is selfish for demanding you pay attention to them first and yourself second. I also do not think that one person in the family can single handedly ruin the family. It takes everyone to do that. Just like a marriage - it takes two people to make the marriage and two people to break the marriage.
I have a toxic member in my family, my brother, who blames everyone else for his problems. I finally had to let him go. He is not good for me and I need to take care of me. I am not responsible for anyone other than myself so....I let him go. I rarely see him. I don't talk to him on the phone but we do have an occasional text. He is gone from my life as far as I am concerned because he is miserable and HE created his own misery. Just like in your family. Each person created their own misery with their own attitude. It sounds like you are trying to leave them behind and they will not let you. But really, it is you who are preventing yourself from leaving them behind. By feeling that you need to see them because "they are family". I often said the same thing but I was wrong. Just because we share blood does not mean we are the best for reach other or that we should change our world for them. My life is so much better now that I have learned this and have left my brother behind. I hope you can come to this place too so you can have a better life.
Take care
P.S. You are NOT worthless. "
Here is my suggestion and comment having read your post, adriasm's post, and Lucky's post.
Number 1: first thing I had to learn in CBT (cognitive behavior therapy) was that the responsibility to change my view belongs to me for life. I'm responsible for myself, no matter what. What does happen when I change the response, the view, the situation changes, sometimes for better, sometimes for more challenging, and then I pause, consider my options, and make changes for my well-being first.
Number 2: where the word "judging" is coming into play, I go to the mirror and have a conversation with myself. I am my shadow, my best health advocate for my well-being. Often, the little voice inside my head changes its words as soon as I start working with the information I have and the choices I need to make for myself to improve my well-being and remove "judging" from the situation.
Number 3: where my feelings, emotions, and actions are adding to the unintended consequences, I put my well-being first. I walk away, pause the response button, and refresh my choices.
Number 4: a decision that I make that I am satisfied with, stays with me, no matter what. What I do change is how I choose to respond to others' reaction to the positive decision I make. Sometimes, this takes a bit of practice in front of the mirror to make myself comfortable with strong statements.
Number 5: I do a handshake to remind myself that sometimes, people have zero clue how to change from being negative to change, rather than positive. I have often found myself just saying thank you and shaking their hand and walking away does more to change the situation than trying to talk about how it works, I just show them how it works for me.
I hope this helps you. When it comes to changing relationships to fit my well-being, I put my well-being first, and then do as much compromising as I can tolerate in family gatherings, and still keep my well-being first. One-to-one relationships that are challenging (or even toxic to my well-being) are kept to a minimum and texting is a positive way to stay in structured communication.
Best wishes. Sometimes, our path gets murky, muddy, and meanders to places we do not expect before we find our inner peace. You are not alone in this.