What do you mean the tenant decides to leave after one day? If they abandoned the premises and stated that they were abandoning the premises, then you should have that in some written format. At that point you'll be able to dispose of the former tenant's property as you wish. But that doesn't seem to be the case.
No, you don't get to withhold property if they don't pay the rent. That is why you want to get a deposit up front from tenants, to ensure that they don't do exactly what they did to you. As a month to month tenant, they have a right to be on the premises for the entire duration of the month to month lease. You'll need to explain what you mean by "decides to leave one day." Ga. Code Ann. §§ 44-7-6 and 44-7-7 should give you a better idea of the laws regarding oral month to month leases and the 30 day requirement for a tenant to terminate the lease. They will owe you money and you will have to sue them in landlord tenant court if the tenant won't pay the rent. Again, this is why it is extremely important to collect an up front deposit to cover your costs.