A judge wont restrict visitation just because the father doesn't visit. Like said above, the father doesn't actually have to visit if he doesn't want to, but we (the custodial parent) have to make sure the children are available to visit or we get in trouble. I went through the same thing with my ex. I too had to take the children to the grandmother's house for the visit and a lot of times he wouldn't even show up. My papers state that if he's not there within 1/2 hour then I can leave. He had supervised visitation then but now he doesn't. He usually calls now if he's not going to get them for the weekend.
Do you have a restraining order on him or is there some other reason he can't come to your house to pick the child up for the visit? If he could do that, then if he didn't show in a 1/2 hour or hour you wouldn't be put out having to drive the baby back and forth. If you have a restraning order or if he has supervised visits then unfortunately you're stuck.
If he does have supervised visits and is not visiting then he is not actually doing the supervised visits. My ex did that too. He had 18 months supervised visits and then moved after 6 months and didn't do any visits. Then came back after 2 years and expected the supervised to be up. He took me to court because I wouldn't let him have the kids unless it was supervised but the judge sided with me because he didn't do the time. He was ordered to finish the supervised visit and told he had to visit or it didn't count.
You definitely need to go to court because it isn't fair that he's wasting your time and the judge will probably figure out another way of you exchanging the child without it being unfair to you, but he wont restrict his visits.
