I dont keep them due to Palytoxin.
It's really the Palys you have to worry about. As long as you are not handling them and then touching your eyes or mouth you should be good to go. In reality you should be wearing gloves and eye protections when handling any coral not just palys. I've definitely irritated my eyes by accidentally touching corals and then my eyes. No fun at all!