Object

Then how do you explain "He was the subject to their affection" Henry?

I don't recognise that as a normal construction. I would say either
He was the subject of their affection, or
He was subject to their affection.
Both carry a similar meaning.