Is there any known magic number signature for CP/M-80 program files?

Question

I'm looking for a way to identify CP/M-80 (i.e. 8080 code) program files (.COM) from their content. A process usually known as Magic Number or File Signature detection, intended to identify a file by looking at their first few bytes for certain values and/or structure.

To narrow this down a bit, it will be fine if the distinction is good enough to separate CP/M-80 .COM files from their counterpart under MS-DOS(*1). That means, while a positive detection of a .COM file being 8080 code is great, already a negative detection of it not being 8086 code will do the trick.

I couldn't find any information. In addition, what I could figure out so far is rather limited:

• CP/M 3 file may start with an optional header starting with a C9h (RET) marking a new, optional header used for RSX linking.

Basic CP/M programs often start with

• Loading the stack pointer LXI SP,const (31 xx xx)
• A variant thereof is LHLD adr/SPHL (2A xx xx F9)
• Some even save SP first with LXI H,0/DAD SP/SHLD xxx/LXI SP,const
• Often this is prefixed by a jump around some header JMP const/....suff.../LXI ... (C3 xx xx ...)

But I also found next to any other instruction when scanning files I have at hand - including pushing all registers onto the OS stack first...

The good part is that on a x86

• C3h is a RET,
• 31h is a XOR r/m16,r16,
• 2Ah is a SUB r8,r/m8

The first is rather useless as a program start, but might be used to make prevent data files from execution, while the second may be unusual, but not unheard of to clear a register during setup. The last one is most definitive unusual for a program start, but as a 2...4 byte instruction any combination may show up as 4th byte, so a check for F9h is even weaker.

So either are rather weak indicators.

---

Question: Is there any (somewhat reliable) fast way for detecting CP/M-80 program files.

---

Background

From comments it seems that it would be of help to add why I'm looking for that information:

I'm in some NEC V20 nostalgia phase, here about the only useful improvement beside speed and 80186 instruction extension: The 8080 emulation mode. I'm tinkering with first drafts of a modified MS-DOS COMMAND.COM that would - or as of now should - detect when a .COM file is selected for execution if that one is possibly an 8080 one for CP/M-80. If this is the case it would

• prepare the PSP according to CP/M (won't need much change),
• add a BDOS to MS-DOS translation stub to top of memory (right below 64KiB),
• Setup emulation vectors
• invoke emulation mode and
• clean up changed vectors after exit.

All of that not a big deal and already working fine, except for the detection part. And yes, I know, this has been done back then several times in form of loaders (which mine already is), but my attempt is to make it seamless, much like it was possible on CP/M-86.

Oh, and all of that should of course work on a 4.77 MHz XT without much delay - remember kids, grampa had to shovel his data with a toy sized scoop, both ways up hill and in the snow.

---

*1 - Using the same extension as CP/M-80 was a real stupid decision. CP/M-86 was at least so kind to go with .CMD - and drop memory images totally.

Answer 1

At user Raffzahn's request, I've expanded a comment into an overly complex statistical answer. In summary, if a random *.COM file is 27392 bytes or fewer and starts with one of twelve byte values, it is very probably a CP/M-80 executable.

There is no practical way to use “magic numbers” to identify CP/M-80 executable files. The Linux file command typically identifies them as one of the following MIME types:

• application/octet-stream
• application/x-dbt
• application/x-dosexec
• application/x-matlab-data
• application/x-tex-tfm
• application/zlib
• image/x-tga
• text/plain

In a review of 5510 unique *.COM files lying about my hard drive (mostly from the Oakland CP/M Archive CD-ROM, the Walnut Creek CP/M CD-ROM and z80pack), with potential MS-DOS or other non-8080-compatible executables weeded out, I found:

Size

• 25% are 1792 bytes or fewer
• 50% are 4864 bytes or fewer
• 75% are 12288 bytes or fewer
• 95% are 27392 bytes or fewer
• All of them are 53504 bytes or fewer.

First byte

• 60% have a first byte (c3)
• 75% have a first byte (c3|21|2a)
• 80% have a first byte (c3|21|2a|31)
• 90% have a first byte (c3|21|2a|31|18|ed|00|cd)
• 95% have a first byte (c3|21|2a|31|18|ed|00|cd|0e|01|11|3a)

MS-DOS

Using all 3508 unique *.COM files on the Power DOS CD ROM (Walnut Creek) (July 1995) and the Simtel MSDOS 1996-09 CDs, I found:

Size

• 25% are 768 bytes or fewer
• 50% are 3072 bytes or fewer
• 75% are 9984 bytes or fewer
• 80% are 12800 bytes or fewer
• 85% are 15872 bytes or fewer
• 90% are 21760 bytes or fewer
• 95% are 32768 bytes or fewer
• All of them are 187904 bytes or fewer. These must have some additional payload to be this large.

(So the typical file size could be smaller under MS-DOS. Perhaps some of these have been packed, a technology that wasn't as effective or widespread under CP/M.)

First byte

• 45% have a first byte (e9)
• 65% have a first byte (e9|eb|8c)
• 75% have a first byte (e9|eb|8c|b4|4d)
• 80% have a first byte (e9|eb|8c|b4|4d|b8)
• 90% have a first byte (e9|eb|8c|b4|4d|b8|e8|fc|24|bc|bb)
• 95% have a first byte (e9|eb|8c|b4|4d|b8|e8|fc|24|bc|bb|be|ba|fb|9c|b0|1e|33)

Note that none of these common MS-DOS first bytes are shared with CP/M-80 *.COM files. Checking the first byte may be all you need to do. Like all statistical assumptions, however, you won't catch every last outlier.

Answer 2

A process usually known as Magic Number or File Signature detection, intended to identify a file by looking at their first few bytes for certain values and/or structure.

As you have found out, there is nothing like a Magic Number like you'd find in an ELF binary or anything like that. Rather, the fact it's executable is determined by CP/M by looking at the filename's extension. Yes, it's another bad idea, but that's how it worked.

Question: Is there any (somewhat reliable) fast way for detecting CP/M-80 program files.

You could look at Markov Chains; essentially read the instruction stream and see which of your models best model your observed state transitions. So then you'd see if your program is an 80186 MS-DOS program, or an CP/M-80 program, or this, or that, or the other. The technique would normally require heavy use of floating point, so maybe not exactly fast if you're intending to do this detection on the V20 itself.

I have no idea if this is a workable idea.

I'm tinkering with first drafts of a modified MS-DOS COMMAND.COM that would - or as of now should - detect when a .COM file is selected for execution if that one is possibly an 8080 one for CP/M-80.

Cool idea! The best of luck with it. Depending on where you want to take this idea, you could consider using some kind of meta-data. See if there's an unused bit in the filesystem, or use a different filename extension, or keep a reference to the executable in another file, or something along those lines.

Or, require the user to explicitly invoke your compatibility layer, just like a Linux user has to type wine minesweeper.exe. Just typing minesweeper.exe on Linux will result in a permissions error.

Using the same extension as CP/M-80 was a real stupid decision. CP/M-86 was at least so kind to go with .CMD - and drop memory images totally.

This part reminded me of a question I asked some years ago which, while being about UNIX and so not directly relating to anything like a solution for you, has some insights of various kinds in the answers. In particular, finding that a binary was for the wrong architecture, I surmise was not a problem that anyone was facing.