Anxiety Support Group
Anxiety is a physical condition marked by intense and persistent feelings of distress, fear, angst or dread. General anxiety caused by routine day-to-day stresses usually passes quickly and is experienced by almost everyone at one time or another. However, such feelings that linger over time and are very difficult to cope with, and which lack a clear cause, may indicate...

If the referral to neurologist and medication would not help? Then you may have only let him see what he knows as neural imbalance or degradation. He only knows what you tell him. The same as any therapist on any level.
If you went in with a set of self diagnosis in mind, he may take that as delusional, which is a symptom of some more severe diagnoses. Ego may also cause him to refer as in "Well you already know so you don't need me, but need help, still"
I know you have already talked about some of your experience here. When I came here a good friend told me to make a timeline of my experiences, just like a health history. Then give it to therapist on visit and ask them if they have experience or are comfortable treating me.
I was referred right away to a more experienced doctor.
It is hard to build a relationship with a doctor if you only tell him/her a part of the story that brings you to them.
Best wishes.