Black sheep of the family
A worthless or disgraced member of a family.
The first record of 'black sheep' in a derogatory sense that I can find in print is from Charles Macklin's The man of the world, a comedy, 1786:
"O, ye villain! you - you - you are a black sheep; and I'll mark you."
It isn't entirely clear why black sheep were selected to symbolize worthlessness. Possibly it is just the linking of black things with bad things, which is a long standing allusion in English texts - black mood, black looks etc. It may also be because shepherds disliked black sheep as their fleeces weren't suitable for dying and so were worth less than those of white sheep.
There is also a contradictory long-standing English country tradition that black sheep are omens of good fortune. The Folk-Lore Record, 1878, included this piece:
"We speak figuratively of the one black sheep that is the cause of sorrow in a family; but in its reality it is regarded by the Sussex shepherd as an omen of good luck to his flock."
Other 19th and 20th century references from Somerset, Kent, and Derbyshire agree with this view that black sheep indicate good luck; others say the opposite. Charles Igglesden, writing on Shropshire beliefs, in or about 1932, gave the opinion that black sheep were considered unlucky and added that the only way to avoid the bad luck is to cut their throats before they can 'baa'.
Old joke alert -
'Why do black sheep eat less than white ones?'.
'There aren't as many of them.'