Heh, I've encountered this before. It's due to a lack of memory that Java cannot access. I suggest examining how much RAM you have and how much memory you are using outside of Minecraft. Then adjust how much memory you can allocate to Minecraft (up to 4G, but that's not usually recommended.)