I would advise you guys to use the word only with someone that you know, is already your friend and you know that he/she won´t take offense.
Don´t use the word with/to people that you don´t know wether he/she will accept. Even if that person lives in Brazil for some time, since that can be deceptive...he/she even can understand our culture and see that we are saying that in a funny/teasing way and meaning no harm.
Even so, this one is one of those things that may depend on individual sensitivity. And even one´s own experience, since even in Brazil the word may or may not have a negative ring to it.
That is, meaning exclusion to a degree - pointing out that someone doesn´t know the language of the country, and is a bit wet behind the ears in someone´s else land etc.
So, if I was from another country I certainly would take it lightly, would accept such word. But I respect their point of view, as per the above reasoning. Instead of expecting that others would share my frame of mind.