A line can be many things - a cord or string, a linear mark, a short letter, a policy, a range of retail goods, and so on. Given that, and the fact that 'hard lines' isn't obviously derived from any one of them, the door is open for the merchants of conjectural etymology.
I ought to mention at the outset that 'hard lines' has no connection with 'taking a hard line', which is an entirely unrelated expression. Clearly the derivation of 'hard lines' is entirely dependent on which line was being referred to when the phrase was coined. There is a reference to lines in the King James Version of the Bible, 1611, and that is the basis of several early citations of 'hard lines':
Psalms 16:6 The lines are fallen unto me in pleasant places; yea, I have a goodly heritage.
'Lines' here is generally interpreted to mean the demarcation of land for the building of a house. In 1866, the American writer John Greenleaf Whittier, in a prose work titled Margaret Smith's Journal, wrote:
My brother's lines have indeed fallen unto him in a pleasant place.
The context of that piece is a woman's admiration her brother's new home and it's clear that it alluded to the earlier biblical phrase.
Another suggested derivation of 'hard lines' is that the lines are those of a ship, i.e. ropes. For an example we can turn to Sir Walter Scott, an inveterate phrase coiner and frequent flyer on these pages, in the novel Redgauntlet, 1824:
The old seaman paused a moment. 'It is hard lines for me', he said, 'to leave your honour in tribulation'.
The nautical association is strengthened by the fact that 'hard line money' was paid to seamen in the 19th century and this was referred to explicitly in the August 1886 edition of the London newspaper The Pall Mall Gazette:
On a Torpedo-boat, besides, there is hard-line money, which makes up for a good many discomforts.
When mentioning anything to do with boats I invariably get correspondence from horny-handed sailors. This one is no exception and several nautical types have informed me that 'hard lying money' was commonly paid to sailors. This was a bonus for those who had to put up with especially uncomfortable berths or other hardships. The suggestion they all make is that 'hard line money' was just a mishearing of 'hard lying money'.
Both of the above explanations make sense, as etymological guesses often do, but neither is the origin of the phrase. The 'lines' in question here are comments written or said about someone and 'hard' in this case just means 'hard/difficult to accept'. 'Hard lines' began to be used in the late 1600s and was a reference to unwelcome disparaging comments. The earliest example I have found is this piece of strangled verse by Mr. John Cleveland, from his Works, 1687:
When sage George Withers and grave William Pryn
Himself might for a poets share put in;
Yet then could write with so much art and skill,
That Rome might envy his Satyrick quill,
And crabbed Persius his hard lines give o'er,
And in disdain beat his brown desk no more.
This is somewhat difficult to interpret but another example comes from The History of France, 1702, in which the meaning is more obvious:
The innocency of the Princes was declared and published, while the Duke on the contrary was detested as an execrable murderer. These were hard lines to the Duke, who wrote his complaints to the King.
The progression from a written or spoken opinion about a person which showed them in a bad light to 'bad luck, old chap' requires no great linguistic leap.
'Hard lines' isn't used so often in everyday speech these days. Fifty years ago, when hearing about a friend's misfortune, we might have bucked them up with 'hard lines chum'; twenty years ago, it might have been 'hard cheese'. These days, the young seem to prefer 'that's a bit harsh'.