I'm not from here!!

One of the many problems with having moved around a lot is conversations like this:

Person: Did you go to this-and-that-event last summer?
Me: No, I just moved here from New York a few months ago.
Person: Oh, you're new here! How do you like it so far?
Me: Well I used to live here before New York, I was only there for two years...
Person: New York must have been very different after having lived here then.
Me: Not really, I've lived in New York before. When I was little...
Person: ...So you are not originally from Florida, but from New York..?
Me: No, Israel.
Person: o_O