Win64 and Linux-x86_64 Calling Convention Unused registers modified or not

Question

I have some important questions about calling convention in linux-x86_64 and win64. I search too many places, but I have not found the answer of my questions !!! I don't think my question is duplicated, so please read it first.

In linux-x86_64 we work with syscalls ...
linux-x86_64 syscall calling convention is:

RDI -> first parameter
RSI -> second parameter
RDX -> third parameter
R10 -> fourth parameter
R8  -> fifth parameter
R9  -> sixth parameter
R11 -> ... (for all syscalls)
RCX -> ... (for all syscalls)
RAX -> return

Now, my questions about linux-x86_64:

Question 1: If one syscall (for example, 'sys_write') takes 3 parameters (RDI,RSI,RDX), what about other parameter registers? Yes this syscall has only 3 parameters but will it use other parameter registers too (for other usage like inside process and ...) ? I mean, if I call sys_write and I have something in R10 register, will R10 value remain 100% unchanged after the syscall ? This syscall has no fourth parameter, so I think everything inside R10 or R8 or R9 will remain unchanged ... correct ? Am I right ?

Question 2: For example, sys_mkdir ... If I have to call sys_mkdir 3 times (one after another), Is this way correct ?

mov eax, 83
mov rdi, .filename
mov esi, 0766o
syscall

mov eax, 83
mov rdi, .filename2
syscall            ; no (mov esi, 0766o) anymore because ESI is equal to 0766o from last syscall

mov eax, 83
mov rdi, .filename3
syscall            ; no (mov esi, 0766o) anymore because ESI is equal to 0766o from last syscall

Here, i just not updated ESI anymore ... since I think syscall keeps parameter registers unchanged. Am I right?

Now Win64, Win64 Calling convention is:

RCX -> first parameter
RDX -> second parameter
R8  -> third parameter
R9  -> fourth parameter
... (Stack)

Question 1: Here, my question about win64 calling convention is the same as the first question about linux-x86_64. if, for example, I call Some function with only 1 argument, (for example ExitProcess) ... will other parameter registers value remains unchanged? Or windows will use other parameter registers too and my value inside them will change ?

For Linux, this is a duplicate of https://stackoverflow.com/questions/2535989/what-are-the-calling-conventions-for-unix-linux-system-calls-and-user-space-f. `syscall` overwrites `rcx` and `r11`, and leaves the return value in `rax`. All other registers, including those in which you passed arguments, remain unchanged. The Windows part is really a separate question and I think it ought to be in a separate post. — Nate Eldredge, Oct 07 '21 at 17:20
Note that (again on Linux) `syscall` has a different convention than ordinary function calls, for which only `rbx, rbp, rsp, r12-r15` remain unchanged and everything else may be overwritten. — Nate Eldredge, Oct 07 '21 at 17:25
@NateEldredge but what about the second question in linux part? — HelloMachine, Oct 07 '21 at 17:31
Read again what I wrote above: " All other registers, **including those in which you passed arguments**, remain unchanged." So in your Question 2, `esi` will keep the value `0766o` throughout the multiple system calls. (Still, readers of your code may not remember that, so it'd be wise to add comments to point it out.) — Nate Eldredge, Oct 07 '21 at 17:33
@NateEldredge true. In windows (x64), it changes the parameter registers values unfortunately (kernel32 functions ...) ... your answer was useful. Thank you — HelloMachine, Oct 07 '21 at 17:36
Better to ask two separate questions in two separate posts -- works better for this Q&A format. — Erik Eidt, Oct 07 '21 at 18:48

Peter Cordes · Accepted Answer · 2021-10-09T20:36:27.320

A registers's status as call-preserved or call-clobbered never depends on the number of args actually passed by the caller and/or expected by the callee, in any calling convention for any ISA I've looked at, and certainly not any of the standard ones on x86.

But yes the calling conventions for raw system calls are different from those for functions, even for presumably thin wrapper functions.

All standard user-space function calling conventions have all the arg-passing registers (and stack slots) as call-clobbered. So if your asm uses call, that's what you need to expect.

The system-calling conventions on mainstream OSes preserves all registers (except the return value). (But on x86-64, only after syscall itself overwrites RCX and R11, because that happens before the kernel gets control.) If you directly use syscall or int 0x80 or whatever, that's what you should expect.

Note that Windows does not have a stable system-call ABI across kernel versions and doesn't document the raw system calls, so in normal Windows code you're always making DLL function calls, never raw system calls. People have reverse-engineered the system calls for different Windows versions, though.

MacOS also doesn't officially have a stable/documented syscall ABI, but in practice Darwin basically does, at least for the normal POSIX open/read/write/close/exit calls that toy programs use.

Win64 and Linux-x86_64 Calling Convention Unused registers modified or not

1 Answers1