My guess would be he saw you hurrying (in his opinion) and asked why in a ham-handed colloquial way, not understanding the implication until the words left his mouth/saw your reaction.
Let it go - you're providing too much rent-free space to a nothing-burger.