Stress
Teaching is a stressful gig. Now add in the fact that you're working with students who have a range of disabilities and you can imagine that the job can get very frustrating at times. Working with these students can take a toll on a person's emotional and even physical well-being, especially when there's constant pressure to make sure students are progressing.
Let's not forget that there are also parents relying on you to make sure that their son or daughter is actually progressing (note: progressing is often subjective and although you may see their child progressing, they might not).
You'll also have to deal with the administration constantly hounding you about new state policy to add such and such in your lesson plans and making sure every kid in your class room can read at so and so level even though they all suffer with varying degrees of disability making it nearly impossible for them all to be on the same level. Just remember: Breathe in...breathe out.
But there will be breaks in the action, so you'll have some time to unwind from everything. You'll get a winter break for the holidays, a week or so during the spring, and the whole summer (unless you opt to teacher summer school, of course).
However, the breaks won't be all fun and games. One of the most crucial parts of becoming a special education teacher is developing each student's Individualized Education Program (IEP), and since special education teachers often work with the same students for an extended period of time, the breaks off are often spent reviewing and updating those plans.
The special education teacher has to take most, if not all, of his/her work home, too. So, while you might be able to make it home in time in order to watch your favorite shows, you may be spending that time grading papers or preparing lesson plans.