I never bought into the "voice assistant" hype the first place. I'd disable the fool thing completely on my phone if I could, or uninstall it. Doing any kind of decent audio processing can certainly be done locally by cell phone hardware, but only by consuming virtually all the CPU and GPU and inference accelerators in the device and maxing out battery usage while doing so.
LLMs are just a poor power/compute efficiency solution for 98% of the things it is being used for - way too many watts burned for far too little useful output. I'd guess LLM "efficiency" compared to hand-written solutions to complex problems to be down in around the 10-25% level at best (gut feel), with 75% or more being overhead for the ineffeciency of the approach.