As it stands right now, you have no rights to the child. Mom can quite literally walk in the front door and demand you hand him back. Not doing so would be a very bad idea.
If the baby is in danger, call Child Services. If not, then sorry, there's nothing you can do.