Off The Cuff

This is an expression I cannot think of any connection to, anyone care to enlighten me?"

Back in the days of separate collars and cuffs, it is alleged that notes for extemporaneous speaking were jotted onto the cuff of the speaker's shirt. Makes sense to me.
Certainly the hand and possibly the cuff as well were handy spots for 'crib notes'.

Why are they crib notes?