By gum

It is known since the early 19th century, as in this example from James Kirke Paulding, in the novel Bucktails, circa 1815:

“By gum, that’s jist what I want you to tell me, I swow.”

‘By gum’ is a forerunner of ‘ee bah gum’ the Yorkshire expression that is still in use in the north of England, although would be considered archaic elsewhere.

This is first recorded as the title of a 1921 song, Eeh! By Gum, It Were a Real Fine Do! by the English vocal duo Kirkby Hudson.