General mayhem; a free-for-all fight.
'Battle royal' may refer to a conflict in which many are involved or be simply a general reference to an intense conflict. In the latter case 'royal' is merely added as an intensifier. Whatever the meaning, there has never been any actual regal involvement implied. The 'royal' is invoked to mean 'a battle fit for a king' as opposed to 'a battle involving a king'.
The term was used particularly to refer to cockfighting, where large numbers of birds were sometimes engaged in 'battle royal' fights to the death. However, the first citations of the phrase don't relate to cockfighting explicitly, so whether the expression originated with cockfighting and then became a more general term for raucous fights isn't clear. The first known record in print is from James Howard's comic play All Mistaken, or the Mad Couple, 1672:
"Hist - now for a battle-royal."
The first citation that refers directly to cockfighting is General Thomas Perronet Thompson's Audi alteram partem, 1857–61:
"Cockerels crow across a ditch, till they get up a battle-royal."