Cutting off contact with the love of your life who's in prison Community Group

Partners in prison who im in love with but have unexpectedly cut off all contact with him. Im his only visitor and hes been really low and i feel like this is the worst thing i could have done, but it was all getting too much for me. Have i done the right thing?