Out of the woodwork

Does anyone know the meaning and/or origin of "out of the woodwork". Does the meaning or origin make it politically incorrect?

"Crawling out of the woodwork" refers to roaches.

I would extend it to any verminous insect or other invertebrate - it is thus used of anyone or anything that is considered unpleasant. I am not aware that there is any political correctness issue.

DFG