Incest Survivors Support Group
Incest refers to any sexual activity between closely related persons that is illegal or socially taboo. Consensual adult incest is very rare. Incest between adults and prepubescent or adolescent children is a form of child sexual abuse that has been shown to be one of the most extreme forms of childhood trauma. If you or a loved one is a survivor of incest, join the group...

I know, being a survivor of sexual abuse, the best thing I ever did was to forgive my abuser. It sounds counter-intuitive, but it really set me free from the prison of pain and heartache and allowed freedom to move on and move forward. Perhaps writing in a journal all of your frustrations and heartache, writing to your brother in there, to at least get it all out even if you never send him a letter or ever confront him, it would at least allow you to get it all out on paper.
As for your SO, if he fully understands and knows your past, but doesn't respect your need for healing and know some days are a struggle, perhaps he's not the "one". True love is gracious and loving at all times, not needy and selfish. So sorry you are struggling. I know prayer really helped me, I pray you will find peace, hope and strength to face each day.