A refusal to accept a proposition - equivalent to 'nothing doing'.
This is a US phrase and originated there in the early 20th century. Gambling with dice was illegal in many states and so gamblers went to some pains to hide the dice when challenged by the police. Courts would sometimes throw out cases if the dice weren't offered in evidence. There are several court records where gamblers were alleged to have swallowed dice to avoid arrest.
The Texas newspaper The Port Arthur Daily News, April 1921, contains the first reference I can find to 'no dice' with the 'nothing doing' meaning:
"It's 'no dice' when the bones can't be found, according to a local court decision. Six white men arrested Tuesday by Officer W D Moore and charged with gaming with dice.
'Did you see them shooting dice?' queried the city attorney.
'No, I did not see the dice.' said the officer."
The men were acquitted. It is very likely that the 'no dice - no conviction' ruling is the source of the current meaning of 'no dice'.
See other phrases that were coined in the USA.