The exception that proves the rule
Normally with these meanings and origins the meaning is well-understood or self-evident and the interesting aspect is how, where and when the phrase originated. This one is a little different - it's the meaning that is generally not understood.
This is one of the most widely misunderstood expressions in all of English.
To the untutored ear it might appear to mean 'if there's a rule and I can find a counter-example to it, then the rule must be true'. This is clearly nonsense; for example, if our rule were 'all birds can fly', the existence of a flightless bird like a penguin hardly proves that rule to be correct. In fact it proves just the opposite.
So, and here the maxim 'a little learning is a dangerous thing' comes into play, it has been suggested that it's an alternative meaning of the word prove that is the source of the confusion. Prove can mean several things, including 'to establish as true' and 'to put to trial or to test'. The second option is what is used in 'proving ground', 'the proof of the pudding is in the eating', etc. It could be argued then that the phrase means 'it is the exception that tests whether the rule is true or not'. In our example the existence of a bird that can't fly would put the 'all birds can fly' rule to the test (and find it wanting).
That's all very well and most people would be happy to stop there. Unfortunately, when we go back to the legal origin of the phrase we see that it doesn't mean that at all. It's the word exception rather than prove that is causing the confusion here. By exception we usually mean 'something unusual, not following a rule'. What it means here though is 'the act of leaving out or ignoring'.
If we have a statement like 'entry is free of charge on Sundays', we can reasonably assume that, as a general rule, entry is charged for. So, from that statement, here's our rule:
You usually have to pay to get in.
The exception on Sunday is demonstrating that the rule exists. It isn't testing whether the incorrect rule 'you have to pay' is true or not, and it certainly isn't proving that incorrect rule to be true.
It's a legal maxim, established in English law in the early 17th century. Written, as law was in those days, in Latin:
Exceptio probat regulam in casibus non exceptis
and is interpreted to mean ‘exception confirms the rule in the cases not excepted’
It has (slightly modified) examples in print going back to at least 1617:
Collins: Indefinites are equivalent to vniversalls especially where one exception being made, it is plaine that all others are thereby cut off, according to the rule Exceptio figit regulam in non exceptis.
While not the earliest citation, this, from Giovanni Torriano's Piazza universale di proverbi italiani, or A Common Place of Italian Proverbs, 1666, expresses the idea clearly:
"The exception gives Authority to the Rule."
See also: the List of Proverbs.