Always the way since cryptography was invented. At some point, as the authorised recipient, you will need to acquire the key to decrypt. By definition, since you have no key at that point, that exchange cannot be encrypted in any meaningful way and is often vulnerable.
Asymmetric key protocols help, but there's still the matter of how you store and retrieve your private key.
 You need a key to decrypt the key and to get that key you need.......oops.
 If you have memorised your 4096 bit key congratulations and get a life.