I wouldn't have expeced otherwise
The Blackphone went down a wrong route. It's just a slightly modified standard phone.
The problem with that is complexity. Mobile operating systems are orders of magnitude to complex to be secure. More complexity means more errors, and more errors mean more security critical errors.
Another problem on those devices is that you have several instances of "binary blobs", code running with very high privileges, facing outside, but having never gone through some sort of security audit.
If you actually want to have a secure device, you need to design it differently. One important thing is to spread out your hardware to different components connected via simple interfaces. Todays mobile phones often have their GSM/UMTS/LTE baseband connected via shared memory or USB, this means that once the baseband is is compromised it's plausible it can attack the application processor, and therefore read out all the keys... or just fake the display.
If you had a simple high speed serial port running a much simpler protocol like PPP, this becomes so hard it gets implausible.
You could have each function of your mobile phone done by an independent microcontroller. The software running on each of those would be simple enough that it would be essentially bug free, so it wouldn't need to be updated. Simple protocols could reduce the attack surface even more.
Without any need to update your software, you could just embed your electronics in transparent resin with a bit of glitter. That would even make the hardware tamper evident.
Then you could greatly simplify the software architecture. Since it'll always be possible to get keys out of your device, and since the CA concept of TLS is severely broken, you could just limit the communication of your device to a single server you own yourself. Since you can exchange the key in advance, you can simply use symmetric encryption. Securing a server is much easier than securing a device that's inside your pocket.