Give up an attempt at something after losing one's nerve.
The UK expression 'bottling out', also called 'bottling it' or 'losing your bottle', appears to have nothing directly to do with bottles. It is widely supposed to derive from the Cockney Rhyming Slang 'bottle and glass', that is, 'arse'. By the way, you might not consider 'glass' and 'arse' to be a rhyme but, of course, as any episode of Eastenders, the BBC soap opera set in East London, will confirm, Cockneys do. The CRS derivation is dubious at best, and I can find no documentary evidence to link it to 'bottle out'.
A more likely source is the earlier slang phrase 'not much bottle', or 'no bottle', that is, 'useless; no good for anything'. This usage is recorded in a glossary of 19th century street English slang The Swell's Night Guide, 1846:
"She thought it would be no bottle, cos her rival could go in a buster."
The phrase continued in use in the 20th century and certainly overlapped with the first known example in print of 'bottle out', that is, the BBC magazine The Listener, March 1979:
"This is the big crime, for them: if they are informers or if they don't have the courage to do a crime. They, as they say, ‘bottle out’.
The use of 'bottle out' hasn't spread greatly outside of the UK. The US still prefers their own version - 'chicken out'.
The phrase has no connection with either 'bottled up', that is, suppressed, or 'bottled off', that is, an unsuccessful performer being hounded off stage by an angry crowd throwing bottles.