I just went through that exercise and found the same thing (kernel source 4.4):
> /usr/src/linux$ find . -name Kconfig | xargs grep "select KEYS"
> ./fs/cifs/Kconfig: select KEYS
> ./fs/ext4/Kconfig: select KEYS
> ./fs/f2fs/Kconfig: select KEYS
> ./fs/nfs/Kconfig: select KEYS
> ./init/Kconfig: select KEYS
> ./net/ceph/Kconfig: select KEYS
> ./net/rxrpc/Kconfig: select KEYS
> ./security/integrity/evm/Kconfig: select KEYS
Looks like selecting any of CIFS, ext4 encryption, f2fs encryption, nfs v4, System Data Verification (whatever that is), Ceph, RxRPC or EVM will select CONFIG_KEYS. Which probably means that, if you build your own kernels, simply turning off CONFIG_KEYS is not an option.