Exclamation of surprise. This is an example of a minced-oath, and is a euphemism for 'By God'.
It is known since the early 19th century, as in this example from James Kirke Paulding (a.k.a. 'Bull-Us, Hector'), 1815:
"By gum, that's jist what I want you to tell me, I swow."
It is still in use in the north of England, although would be considered archaic elsewhere.