Joint Bank Account Dispute

My fiance and I opened a joined checking/savings account with Bank of America while we were living together in Massachusetts. We both moved back to Michigan, and then a few months later he went back to Mass and I stayed here. A couple months after that he cleaned out the bank account (though most (not all) of the money was from his paycheck) and closed the account without my permission or knowledge.
I went to check on my new BOA account online, and noticed that the account had been reopened, hosting both my direct deposit and his. I transferred the funds in their entirety to my account, and closed the account back down, with every intention of sending him a money order for the amount of his check. This morning, before I had a chance to call him and let him know (I work midnights so I get up a little later than most) he and his mother are calling me threatening to put me in jail for grand theft etc. Assuming I didn't send the money (though, honestly even still I likely will), would I not have rights to the money, as it is in fact a joint account? The amount was $550.00. How would being in the two different states affect that? If he did file claim, would I have to go there, or would he have to come here?