Who is the children's "legal" Father? This is important if the children were concieved during your marriage he is the "presumed" Father and would have rights for visits.
We had not even met when my children were born and their real father has visitation but refuses to see them. I met my husband when my children were 4yrs old and 1yrs old.
Then he has no rights to visitation however if the children have a bond with him and he wishes to keep that bond in tact. Especially with Bio Dad not around its possible a judge could grant visitation. You should discuss this with your Attorney
How old are the kids now? Since the kids have no active bio dad in their life, you need to decide if cutting off all contact from their step dad is really in their best interest. True he has no legal right to them, but you need to start thinking about what is best for them.