Yeah, I used to be able to jot down 6502 hex codes directly as well.
For Z80, iAPX86, 68K and DEC I needed an assembler.
The x86 was horrible, from a programmers point of view. 6502 and 68K were much more aimed at assembler / machine code programmers. I always had the feeling that the iAPX86 was designed for high level languages and was downright hostile to machine code programmers.