A prize given to make fun of the loser in a contest or game.
A Booby is a type of gannet. 'Boobies' has also been used as a slang term for breasts since around 1935. This booby is neither of those. The word has been used to mean dunce or nincompoop since at least the late 16th century and that's the 'booby' of 'booby prize' and 'booby trap'. The word probably derives from the Spanish word 'bobo' meaning 'fool' or 'dunce'. An example of that usage, as 'bobie', comes in The Pleasant Comodie of Patient Grissill, 1599:
Let Gwenthyan see what bobie fool loves her...
It is also found, as 'booby', in Fletcher and Massinger's comic play The Custome of the Countrey, circa 1640:
Cry you great booby.
Some etymological records have it that people were labelled as boobies because they were like the seemingly dim-witted seabirds. Actually, it was the other way about. The blue-footed avians were named 'boobies' by the 17th century sailors who first came across them in the eastern Pacific because they didn't attempt to escape when approached. Thomas Herbert recorded this apparently stupid and 'booby-like' behaviour in the travelogue Relation Travails, 1634:
One of the Saylers espying a Bird fitly called a Booby, hee mounted to the top-mast and tooke her. The foolish quality of which Bird is to sit still, not valuing danger.
The phrase 'booby prize' originated as a term of disparagement for the person with the lowest score in a contest. It was probably coined in the USA - all the early printed references to 'booby prize' originate from there. The earliest that I can find is from a student newspaper from Oberlin, Ohio, The Oberlin Review, 1881:
The gentlemen strove at skill in hemming aprons for the fair ones: the first prize was won by Mr. Jackson, the booby prize by Mr. Jones.
See also - 'booby trap'.