Our right to freedom of speech means, among other things, that the government may not punish us for insulting others or calling them offensive names.
That said, had you responded that way you may have goaded him into attacking you. While the city could prosecute the guy for doing that, that's...