At the Drop of the Hat

Since the earliest examples of this phrase in print have no specific mention of fighting, my bet is on the start-of-a-race explanation from James Rogers' Dictionary of Cliches:

Dear you, Thanks for Visiting Brahmins Net!
JaiHind! Feel free to post whatever you think useful, legal or humer! Click here to Invite Friends

DROP OF A HAT - Acting readily or on some single signal. In the 19th century it was occasionally the practice in the United States to signal the start of a fight or a race by dropping a hat or sweeping it downward while holding it in the hand. The quick response to the signal found its way into the language for any action that begins quickly without much need for prompting.

The earliest reference was from a hearing on a bankruptcy law from an 1837 Register of Debates in Congress