CP == Hypervisor
In IBM's VM/370, CP is the Control Program, which we now call the Hypervisor. It was responsible for creating and running the various guest operating systems. Since every guest operating system up to that point was definitively not personal in any way (think monster OS/VS1 and bigger monster MVS), the nice folks in IBM Cambridge also developed CMS, the Conversational Monitor System. (It was first called the Cambridge Monitor System but IBM marketing said no sorry). This was an IPL-able (aka bootable) OS that provided a facsimile of a "personal mainframe", which you could use to create and edit files, run some utilities, and interact in a limited way with other CMS users. CMS would not run on a bare metal machine - it required a VM/370 CP environment.
The # character (that would be an EBCDIC # thank you) was the default escape character used to signal that the command should be intercepted by the CP rather than the guest OS. when that became necessary, eg to re-IPL. Another way to escape was to hit the 3270 "PA1" key, which would then normally display a status of "CP READ" allowing direct input of CP commands. Just to muddy things a little more, the CP could receive commands directly from CMS as a courtesy default, meaning the #CP prefix wasn't necessarily always strictly required under CMS.
It was common as dirt to run VM/370 as a guest OS under VM/370. There are likely few VM system programmers from that era who haven't done it just for fun but it also served a real purpose: testing new versions of VM/370, testing new important guest operating systems under new versions of VM/370 and testing local modifications to VM/370 were all useful tools. When running VM under VM (under VM etc), the escape character would be different (it was settable as well). When juggling CPs, one was well advised to keep track of one's place on the perilously teetering stack of operating systems and issue commands to the proper CP using the proper escape character (EBCDIC please). Someone not used to this juggling might reflexively type, eg #CP IPL because that's what motor memory has instilled, but such haste can easily cause the teetering stack to vanish in the click of an ENTER key. Oopsie. Don't do that.
The NYC/northern NJ computing landscape of the late 1970's and early 1980's was littered with IBM mainframe shops running VM/370, MVS, VS1 and DOS/VS. There was more mainframe iron there than anyplace else on the planet, it seemed, and everyone who was anyone in the VM world met up at the MVMUA conferences, usually held at MetLife in Manhattan.
-Grayhaired VM/370 systems programmer
Biting the hand that feeds IT
