Re: It doesn't matter that it doesn't relocate in RAM while running
First of all, the actual ASLR entropy is much lower than that in any sane implementation, for various reasons. Usually it's a couple of address bits.
Second of all, often you can figure out the ASLR base from a single leaked pointer. Then you have effectively defeated ASLR.
See https://grsecurity.net/~spender/exploits/wait_for_kaslr_to_be_effective.c for an example of how this is done against Linux.
This is one of the reasons why you should always build a custom kernel when security matters, and protect the build tree and kernel image itself from potential intruders. That way just leaking the kernel base is not enough - you still have no idea of exactly where specific code and data lives inside it.