You do not HAVE to allow them to speak with your son. However, that may not deter them from proceeding with whatever action they might choose to take.
Two children at ages 5 and 3 playing a game of "you show me yours, and I'll show you mine," is not a huge problem and not likely to get the authorities all that excited. They were far too young for any crime to have occurred, and chances are memories will be sufficiently foggy that nothing would come of it anyway.
While you have the right to deny access to your child, that might serve only to prolong the process and the investigation. I suspect that child services merely wants to close the matter out, and the quickest way to do that is to talk the child. if they can't, it may remain open for a long time to come.
If you are truly concerned, and afraid there might be more to all of this, you might consider consulting legal counsel.
