You do not HAVE to allow them to speak with your son. However, that may not deter them from proceeding with whatever action they might choose to take.
Two children at ages 5 and 3 playing a game of "you show me yours, and I'll show you mine," is not a huge problem and not likely to get the...