One Belligerent Old Fuck Idiot

Makes my life a living hell last night on security post.  He was not on the resident's guest list, and, when I called the house, nobody answered, so I told him he's going to have to call the homeowner to call me to authorize his entry.  He couldn't hear very well, and got quite belligerent, saying "no, I'm not gonna call her" and "no, I'm not on the goddamn guest list", said "the woman living there is 88 years old so of course she's not answering the phone" and said "open the goddamn gate".  After we had words, I stupidly admitted him.  How the hell do I know who he was and what he was doing there?  Is he her son?  Her caregiver?  A thief?  A murderer?  I shouldn't have admitted him, and should've called the cops to escort him out of there.  I hope this doesn't come back and bite me in the ass.
Needless to say, this job sucks.  I don't wanna do this anymore (I never did).  Hope this year's video season picks up, or I can find something else to replace this nonsense, this debacle.