A much vaunted discovery, which later turns out to be illusory or worthless.
There are two unrelated meanings of 'mare's nest' in circulation, and there's little to connect them. The first, and 'proper' meaning, has it that finding a mare's nest is imagining that one has found something remarkable when in fact one has found nothing of the sort. The second meaning, which is more widespread today, is that a mare's nest is a confused mess - more on that later.
The earlier 'misconception' meaning has been in use since at least the 16th century, when Robert Peterson published a version of the Italian John Della Casa's Galateo. This was 'done into English', that is, translated, by Peterson in 1576:
Nor Stare in a mans face, as if he had spied a mares nest.
Animals are often alluded to in phrases of this sort, for example, lion's share, dog's breakfast, bird's-eye view etc. Of course, this one is different, in that mares don't make nests - the allusion was meant to be comically ironic. That humour is reflected in several of the early citations of 'mare's nest' (or horse's nest, as some early references have it), which refer directly to laughter, for example, John Fletcher's Jacobean tragedy Bonduca, circa. 1613
Why dost thou laugh? What Mares nest hast thou found?
The joke was pushed further by Dr. [Jonathan] Swift, in the play Miscellanies, 1751:
What! Have you found a mare's nest, and laugh at the eggs?
Back to the second, 'muddle' meaning, which didn't begin to be used until the mid-19th century. It appears to have come into being as the result of a simple misunderstanding. To someone who was unfamiliar with the original meaning, and that meaning is hardly intuitive, 'a mare's nest' would seem very much like the earlier 19th century phrase 'a rat's nest'. In reality, rats make rather neat nests, but the phrase was certainly meant to mean a disordered tangle (see also haywire) and the currently widespread meaning of 'mare's nest' was copied from that.
The transition from the earlier meaning to the later one was gradual and appears to have been well underway by the 1920s, when Agatha Christie wrote The Mysterious Affair at Styles. Christie managed there to use both meanings in the same story:
a misunderstanding... "In my opinion the whole thing is a mare's nest of Bauerstein's! ... Bauerstein's got a bee in his bonnet. Poisons are his hobby, so of course he sees them everywhere."
and, a muddle... "A pretty mare's nest arresting him would have been."