Put it this way:
NOTHING you have said proves Dad unfit when it comes to custody.
Yes, you chose to do what you did. You didn't exercise due diligence in choosing a partner. This is NOT all his fault.
With regards to him being unfit, you need to forget that angle. Is he dangling the child over boiling oil every evening? Exposing the child to a child molester on a regular basis? Driving drunk with the child present on a regular basis? Feeding the child beer and special brownies? Not feeding the child?
You would be astonished at the "acceptable" standard of parenting.
Please, talk to an attorney if you don't like what you're being told here.