Unless the children were born during a marriage, he's not legally the father.
If he's not legally the father, he has no parental rights UNLESS he goes to court, has DNA testing done, and the DNA tests reveal him to be the father.
After that is done, however, he has to get the court to create a custody/visitation order, and a child support order.
How does that apply to you?
If the kids were born to him without you being married, he has nothing to relinquish, as he's not legally (nor has he ever been) their father!
Legally, all of the parental rights, responsibilities, and obligations for the children are
yours, alone, mother.
If that is the case, stop communicating the the deadbeat, lout.
Keep him away from your kids.
Think of it this way, ma'am, I appear and say your kids are my kids, too.
After laughing at me, you'd call the police and report a maniac is trying to take your kids.
I'm a legal stranger to your kids, as is he.