Physical Danger
Patrolling the border can be an extremely dangerous job. You've got to worry about desperate people trying to cross the border, drug cartels, and even hungry coyotes out for a midnight snack. And most of those coyotes are not carrying valid documentation.
Agents carry guns, and so do some of the people they're looking for. Drug dealers and other criminals tend to take personal security very seriously. If someone wants to hurt you bad enough, even a rock can be a deadly weapon.
Speaking of which, there's the borderlands themselves. Whether high in the Rockies or way down in the desert, anything from weather conditions and temperature to wildlife or a sudden drop down a cliff you didn't see can mean the difference between patrolling and never walking again.
If you're afraid of getting hurt, this isn't the job for you. To be a Border Patrol Agent, your actions can't be based on the fear of being hurt or killed; this is doubly true when dealing with other people. Staying safe is important, but avoiding any and all risk means you won't be able to perform your job to the greatest extent of your ability. Don't be reckless, but don't be so afraid you never even see a foreigner.