in my experience the pcb part of the battery pack is designed as a safety feature to cut the power if the pack falls below a certain voltage, thus if the output voltage drops for any reason,below this it cuts out, it seems to me that either the load on the battery is pulling the voltage below the cutoff, or the batteries are not taking enough charge to keep the voltage high for long.
in such a small space you will find it difficult to get much better than lipo, for capacity vs weight and size. you can get the lipo packs in all shapes and sizes, just get a suitably sized pack with a connector and balance lead and a charger and you should be good. it might be worth trying to measure the current draw when operational, or check the voltage when various components are run, to get an idea of the reason the battery trips out. I tend to make up a lot of my own battery packs from 18650 cells, and add the balance leads, if you do this without the pcb you do have to keep an eye on the voltage so I use a plug in battery monitor that connects to the balance lead and gives an audible warning when the cutoff voltage is reached, but that would lose you space in your battery compartment, its not huge about 25mm square and 6mm thick.
