The natives are getting restless

Hey,

I'd like to know what is the origin (and the meaning) of the phrase "the natives are getting restless". I believe it dates to the colonial times of British empire. But what sort of context can it be used nowadays?

Thank you,

Mikko Viljanen

I believe this phrase is now considered "politically incorrect" as it is somewhat derogatory.