Rock Pillars can block exit to the surface

Please check screenshot - should be obvious. I went down to caves from big F bird island and then I can't go back without destroying the pillar. Destroyed rock pillar still blocks, which means cave entrance is one-way trip and I can't revisit island without bird's help.