This phrase is widely thought to have been taken from the scoring in card games like cribbage (a.k.a. crib), in which the score is kept on a pegboard. The game ends when the first person 'pegs out'. That view may be correct, but there doesn't appear to be any conclusive evidence to support it.
That use of 'peg out' in card playing is first recorded in Hardy & Ware's The Modern Hoyle, 1870:
"He may with a very poor hand be just able to 'show' or peg out."
Cribbage is quite an old game and John Aubrey in his biographical work Brief Lives, said that Sir John Suckling, (1609 - 1641?) "invented the game of cribbidge". This seems to be a reliable account and is supported by other contemporary evidence. It may be then that the term 'peg out' was used prior to 1870 but wasn't recorded.
Whether or not 'peg out' was used in the card game, it isn't conclusive that this usage is the origin of the 'to die' meaning. That figurative use is first seen in the USA in the 1850s, but no early citations of its use with that meaning refer to cards. The first record of it that I've found is in J. M. Field's Job & his Children in America's Lost Plays, 1852:
"To think what a blessed mess of piety one's got into, and 'bleeged to keep it up until Daddy Day pegs out."
The term was obligingly defined in the Ohio newspaper The Marysville Tribune, in October 1855:
"Peg out" is a slang phrase for die."
See other phrases that were coined in the USA.