Your custody issue is a problem. Unless you can prove - and by prove, I mean police reports, convictions, etc - that he's unfit, Dad is going to get unsupervised visitation probably from the get-go.
You can make all sorts of claims, but the bottom line is that he's going to get some...