I am resistant to telephone visits

I'm wondering if this is a HSP trait. After I have worked all day, I don't seem to have the energy to return telephone calls by friends or family who want to "visit". I end up being irritated and annoyed that they want to call during the week when I have been so busy with working, errands, meetings, work-outs and extra-curricular activities. By the time all of this is done, or even in the midst of it, the last thing I want to do is get on the telephone and "catch up" with people. On weekends, I'm on the go so much that it seems difficult to find the time to stop long enough to catch up with telephone visits. By the time I do stop, I just want to rest.
These telephone visits are with family or friends who live long distance from me.
Can anyone relate? Does anyone have any suggestions?