The Father or presumed Father if not legally established can say whatever he wishes. It sup to a court to decide and that decision will be based on best interest of the child. Be ready to show court why child is better off with you. Do not make accusation you cannot prove though