Visitation isn't taking away any right of yours. If there is a sibling involved, it could be and often is beneficial for them to be treated the same. This guy was essentially her father for as long as she remembers. She is only 5 or 6 now if you were together 4 years and she was 1 when you got together, and that is far from old enough to be able to call the shots on visitation. This guy did raise her for 4 years and is the father of her brother. If you have significant concerns about the children's safety and well being, by all means file for a modification of visitation for your son's sake. If this is just a matter of you not wanting to cooperate with an ex, you might reconsider voluntarily allowing her to visit with her brother on occasion. I know many families with a stepchild who spends time with a half sibling and the half sibling's parent to support the sibling bond. It may not be ordered if this guy is not the biological father but you might want to consider it.