If you go into court sounding like you want to punish dad for "abandoning" you by limiting contact, it won't go well. Judges tend to favor parents who provide primary care AND who facilitate the other parent's involvement when deciding on where the child will primarily reside.
Something much more detrimental that what you've mentioned would likely have had to occur for dad to get limited visitation at this point. Simply leaving and then coming back could be construed as a vacation, he had to work, he needed a break from you, he had a sick mother, who knows.
And no, four year olds don't get to decide where they live, they're four. In my opinion, if dad wants to reunite with his child at some point, your job as mom is to help your son see that as a good thing. After three months, your son still knows who dad is. Limited, supervised or graduated visitation is for abusive parents or for kids who don't know who their parents are, not for little boys who don't need any more barriers between them and their dad.
Does it suck? Yes. Will it break your heart? Absolutely. But it must be done. If a much longer time goes by and he never shows up, you'll have a different problem and it won't get any easier on your son.