"a huge performance hit"
Moving a few megabytes of memory, and updating some pointers? You do realise that modern CPU's run at multiple GHz? And move memory at many 10's of GB/second? Stop talking bollocks. Grab the kernel mutex, move everything, 0.001 seconds later it's done, release it. You won't even notice.
If you're ever inclined, boot into VGA or a VESA mode on a modern machine and drag a few windows around. See how much you don't notice moving a few megabytes around. It's not as quick as you think.