If you were never married to him, you already have sole custody. It's up to him to establish his rights (paternity, visitation, and support obligation). You don't have to allow contact without a court order.
But, if he files in court, he will get visitation. Your standards for a residence are...