Man of the cloth

What's the origin of this phrase that refers to a member of the clergy? My husband is one and we have never found the origin of "man of the cloth." Thanks.

From Morris Dictionary of Word and Phrase Origins by William and Mary Morris (HarperCollins, New York, 1977, 1988):

"man of the cloth was originally a term applied to anyone who wore a uniform or livery in his work. A baker's white jacket and trousers would be called 'the baker's cloth.' But by the seventeenth century, 'man of the cloth' came to be restricted to the clergy and the expression 'the cloth' meant clergymen collectively."