Incest Survivors Support Group
Incest refers to any sexual activity between closely related persons that is illegal or socially taboo. Consensual adult incest is very rare. Incest between adults and prepubescent or adolescent children is a form of child sexual abuse that has been shown to be one of the most extreme forms of childhood trauma. If you or a loved one is a survivor of incest, join the group...

if you haven't already, I think you should talk to him and set some behaviour rules that are non-negotiable. Don't justify, explain or apologise. Just tell him these are the rules of your house and that everyone has to stick by them. end of.