The father needs to go to court to get visitation for the child. Unless there is something in writing from the courts then the guardian does not have to allow visits unfortunately.
Was the father and this woman ever married? Was paternity established?