This literally happens with every single world I've ever played, going at least as far back as the Cosmic Upgrade. It might be an integrated video thing, it might be a Linux thing, but it's definitely not a "some plants" thing.
edit: To be fair, results are worse in my current Arboria map (3-5 seconds to enter) vs an old CU map (1 second to enter, but still perceptibly slower than the "plant growth stage" indicator behind F10) vs a QoL3 map (1.5 seconds to enter). But it wasn't good before.