Should I feel bad?

My brother is upset with me because i said I didn't care how he felt. I don't think what is he going to feel about what I am doing.
I don't feel concern for other's feelings. I do what I think is right and so I don't think I am hurting anyone or causing them any problems. If I am doing what I feel is right then it shouldn't hurt anyone's feelings. I can make a mistake or access something wrong and a person may have their feelings hurt but I don't know how thinking about their feelings before hand would have caused me to act any different if I was doing what I thought was right in the first place. I'd spend time thinking of everyone's feelings and then decide what is right and more than likely make the same decision.
Acting this way I think that if I do something that bothers them then it is them not me that needs to deal with their feelings. I don't mean to hurt anyone's feelings. I don't really think overly much about other's feelings one way or the other unless I am trying to make them feel better or it is their feelings that is the problem with which I am dealing.
But my brother thinks I should care about his feelings and love him only because he is my brother and that I should be different with him than with others. Maybe he is right. I think I don't because he seems to have shown me little respect, friendship or favoritism in the past and the only things he does for me is because he feels he has to or is responsible to do because he is my brother not out of love. I respect that he has this sense of responsibility or care for me because he is my big brother but I don't see any reason to reciprocate a false feeling for him just because he is my brother. I dont do anything that is wrong to him and try to always do what I think is right with everyone so what does it matter if I dont care for him or feel for him more than I would anyone else?