The size of the SPI_Flash memory is 8MB so you must take care of the packages you are going to install. The perl module uses a lot of modules and packages. Have you tried to use only perl without perl-base? Are you adding more packages or not?
I tried it using both:
IMAGE_INSTALL_append = " perl"
IMAGE_INSTALL_append = " perl-base"
It will compile ok just simply using perl, but then the image is too big to use. My assumption was the perl-base was the smallest possible version of perl without any of the other fluff. But I'm still not sure if that's even the right way to add perl to my image