Or do they? Certainly that seems true for illusions that rely on low-level sensory processing. But what about illusions that require more interpretation from our brain? People have different impressions from color illusions like the blue dress, and hear different things in the Yanni/Laurel recording. Some visual illusions don’t work on babies, because they haven’t yet learned to recognize certain cues in their environment. The family gathering illusion is like that — it depends on your experience. We call it an illusion because your brain makes its assumptions automatically, without you even realizing this interpretation is taking place. There’s also some research that suggests that non-industrialized cultures might not be fooled by some of our favorite illusions (like the one with the arrow-tipped sticks that appear to have different lengths).
That said, it’s true that there isn’t a scientific definition of the word illusion, and we use it to refer to all kinds of very different phenomena.