I raised an Asperger's Child (High Functioning Autism) and a Severely ADHD Child (both with other disabilities -- but also testing as highly Gifted) -- and it is a YES and NO Answer --
SOME Think that because a child is Different -- they can NOT hear when you whisper about them -- and that is RUDE - because most of the time the CHILD (themselves) will pipe up and tell you in no uncertain terms what they perceive is behind the whispering -- and I KNOW individuals who have taught my children, did whisper -- and did it with the idea (in only their minds) that it was a subtle way of trying to tell me to REMOVE my child from the programs that they were qualified to attend!
Then there are the Loud and RUDE ones, who let everyone in the area know that they DO NOT WANT your disabled child in THEIR space or Classroom or the Event or whatever -- these individuals are AFRAID of anything DIFFERENT than they are -- so want to have the disabled OUT OF SIGHT, Out of Mind (according to them).
EITHER Way -- I, the Long-Term Single parent of these disabled children (now adults and living independently -- but with help), ALWAYS stood up against the prejudice and ignorance of others -- and did my best to EDUCATE them that just because someone is differently abled (and I am the child of a Paraplegic WWII Vet -- who was in their early 40s when they had me and my siblings) -- and it just is plain aggravating that they would make comments because someone is in a wheelchair, or walks with canes, or needs crutches, whatever. Always growing up with a Disabled Vet of a Father -- I can say that I definitely see things differently than others -- and NEVER let anyone with any disability feel less than welcome wherever I am at.