I doubt you have abandonment issues. Usually, it's six moths plus of not paying, and no visitations.
Have you attempted to get an adoption package from your state? I would start that. If he's got money, let him fight that, if you lose, then fight custody.
I think you are setting very well...