Physical Danger
You'll be in courtrooms and in prisons a lot. They're actually pretty safe with all the guards, security checkpoints, and steel bars. As a defender, you won't have to worry about a vengeful gang leader seeking revenge after you put him behind bars. You can, however, suffer the wrath of angry victims: victims of crimes committed by the person you're now obligated to defend.
Rationality goes out the window in these situations. No one wants to hear about "due process" or "The Bill of Rights." All they see is you sticking up for the jerk who robbed their house, held them at gunpoint, or killed their son. Beware. You're working in a world filled with anger, sadness, addiction, and crime. Be ready for anything. And always wear a tie.