Originally a bag or sack with dirt in it. More recently an unkempt or slovenly person.
The sack of dirt meaning is American in origin and dates back to the 19th century. That's first cited in the Iowa newspaper the Davenport Daily Republican, 1897:
"Hundreds of section hands are striving to keep the water back with dirt bags."
That reference is to what would be called 'sand bags' in the UK. 'Dirt bag' is also the commonly used term for a bag in a vacuum cleaner that holds whatever is sucked up.
'Dirt bag' in the more recent 'unkempt/unwashed' sense clearly alludes to the actual bags of dirt. The first printed reference to it is in an article in Mansfield News Journal, May 1976 which comments on the case of Spider Sabich, who was shot by Claudine Longet:
"Spider, the object of constant adulation, persistent attentions, used to say, "I'm just a dirt bag. Who am I trying to fool?" ... She [Claudine Longet] didn't consider herself a dirt bag."
The term was given a boost when the band Wheatus released a single titled Teenage Dirtbag in 2000.