An exclamation of surprise.
It is commonly thought that this expression refers to James Gordon Bennett. JGB was a real person - in fact, with the expansiveness that is appropriate for this story, two real people. The elder James Gordon Bennett was born in Banffshire, Scotland in 1795 and emigrated to the USA, eventually becoming a journalist and founding the New York Herald in 1835. Bennett had a natural talent for journalism and the paper flourished. An editorial in Harper's at the time expressed the opinion that "It is impossible any longer to deny that the [city's] chief newspaper is the New York Herald". Other rivals, while accepting Bennett's nose for a story, weren't impressed with what they saw as his 'gutter press' methods. In 1836, in a pre-cursor to the chequebook/kiss-and-tell journalism now so popular with tabloid newspapers, he published a notice offering to reward any woman who "will set a trap for a Presbyterian parson, and catch one of them flagrante delicito [sic]". He was unblushing in what was then seen as improper descriptions of his relationship with his wife - describing her 'most magnificent' figure and publishing details of their wedding and the birth of James Gordon Bennett junior in 1841.
James Gordon Bennett Jr. inherited his father's talents for journalism and controversy, not to mention his multi-million dollar estate - and he's the Gordon Bennett that the phrase refers to. He took over control of the New York Herald in 1866, by which time he was well into an enthusiastic and hedonist playboy lifestyle, indulging in spending the family fortune on air and road racing in the USA, England and France.
He was a significant promoter and patron of sports, especially those requiring impressive and expensive equipment, for example international motor racing, ballooning and air racing. He gave several sponsorships in these fields, notably the Isle of Man Bennett Trophy races of 1900 to 1905 (subsequently a trials course on the island was named after him). A long-distance hot-air balloon race (The International Gordon Bennett balloon race), which still continues, was inaugurated by him in 1906.
Bennett was also a chip off the old block, not unlike many wealthy people of his era, in that he wasn't especially concerned by people's opinion of his behaviour. He has the unenviable record, as bestowed by the Guinness Book of World Records, of the 'Greatest Engagement Faux Pas', for the manner in which his engagement to the socialite Caroline May was broken off in 1877. The engagement was big news in New York society circles. The Edwardsville Intelligencer, reported it in November 1876:
"The trousseau of Miss May, who is to marry James Gordon Bennett, has arrived from Europe, where it was collected at an expense of $20,000, according to gossips. It is said to be the most elaborate and beautiful ever prepared for an American lady."
It is reported that at the 1877 New Year's party held by his fiancee's father, he became so drunk that he mistook the fireplace for a toilet and urinated in it in front of his hosts and their guests. Whether or not that story is true is now difficult to verify. It is certainly the case that the marriage didn't go ahead and that the Mays weren't best pleased with Bennett - as this piece from The Perry Chief, January 1877, indicates:
"James Gordon Bennett was publicly horse-whipped this morning, by Frederick May, brother of the girl to whom Bennett was engaged to be married."
He took to his heels and travelled to England, ending up in Melton Mowbray. Perhaps he had heard of the town's paint the town red story and thought he would be at home there? Even the thick-skinned Bennett had the wind taken out of his sails by these events and he remained single until he was 73, when he married the Baroness de Reuter.
There are many other stories listing his excessive and occasionally boorish exploits. These didn't stop him being an successful and innovative journalist though. He invested heavily in developing on his father's news empire. In 1868, with the simple brief of 'find Livingstone' he sent the travelling correspondent of the New York Herald - Henry Morton Stanley, to track down and interview David Livingstone in Africa. After a long search Stanley was ready to give up but was encouraged by Bennett which, when he eventually located his prey on the shores of Lake Tanganyika, resulted in what has become one of the most famous of all journalistic lines - "Dr. Livingstone, I presume?"
From 1877 Bennett lived in Europe and continued to run the New York Herald from his $600,000 314-foot yacht, the Lysistrata. He died in 1918.
The expletive Gordon Bennett appears to be a minced oath. It is a version of Gor blimey, which is itself a euphemistic version of God blind me. That, combined with Bennett's famously outrageous lifestyle and newsworthy stunts, is sufficient to explain why his name was picked out.
That's why; so what about when? The name Gordon Bennett appears in print many times in the 19th century, as we might expect of such a newsworthy figure. The earliest example that I've found of the expression being used as an expletive is in a novel by James Curtis from 1937 - You're in the Racket Too:
"He stretched and yawned. Gordon Bennett, he wasn't half tired."