I think things of that nature should have been covered in the original agreement between the parties, concerning visitation, vacations, etc.
If there is nothing in stone saying you must contact before travel, I don't think you should worry about it. It is, afterall, your...