Ich is a parasite. Velvet is a parasite. Flukes are parasites. All these have symptoms that most people would see as negative: torn fins, cloudy eyes, excess mucus, etc. Even when fish become partially immune, they are still being affected by the parasites. If they are living in an aquarium that is infected, they will constantly be infected by the parasites. They may have resulting infections. Some fish may develop a stronger immunity, but some fish may develop very little. You have no idea how being constantly infected may affect their health and behavior versus if they weren’t.
You're simply wrong.