There's "physical access" and "physical access"
Sure, if you are alone in a room for a long time with all the needed tools, you can have full physical access, you can disassemble a machine and re-assemble it later, just, if that is one of the glue-assembled ones you need some specific tools and time to take it apart, and put it together again.
But a bug that needs just a few keystrokes to be exploited, doesn't require much "physical access", you just need a few minutes to type on the keyboard - a very different kind of "physical access".
And Macs are not usually machines buried deep in some remote server room...