Then you remove the whole plug, not the stem.Problem is that sometimes those plugs bring in nasties like some types of algae.
I never leave them on anymore.
And I don't think that was the reason OP attempted to remove it in the first place.
Also, IMO algae isn't what anyone should be afraid of introducing, it'll get there anyway if it isn't there already.
Only reason to be afraid of plugs are pests, which can be examined using a magnifying glass and proper treatment - though obviously, when in doubt you always need to remount.