I am confused about those caller-saved and callee-saved registers. What if a function is both caller and callee?
Say the main function calls function P, and function P calls function Q. In this case, P being both callee (for main) and caller (for Q), then which register will the assembly use?
For the sake of simplicity, I will assume the System V ABI, and limit this explanation for integer and pointer parameters. For a much more detailed description of the calling conventions for Unix refer to this post.
The first six arguments are passed to the function in the registers rdi, rsi, rdx, rcx, r8 and r9. These registers are caller-saved, i.e., they may not be preserved between function calls, since they callee may clobber them (a more suitable term for them would be call-clobbered registers).
Let's declare Q as:
int Q(int, int, int, int, int, int);
That is, Q takes six integers so that the compiler-generated code is forced to use all those six caller-saved registers.
Now, let's define P in this way:
int P(int a, int b, int c, int d, int e, int f) {
return Q(a, b, c, d, e, f) + Q(f, e, d, c, b, a);
}
This way the compiler will run out of caller-saved registers, and it will be forced to use callee-saved registers for the code of the function above.
The code above generates the following assembly:
P:
// save rbx, rbp, r12, r13, r14 and r15 onto the stack
// copy rdi, rsi, rdx, rcx, r8 and r9 into those registers
pushq %r15
movl %esi, %r15d
pushq %r14
movl %edx, %r14d
pushq %r13
movl %ecx, %r13d
pushq %r12
movl %r8d, %r12d
pushq %rbp
movl %r9d, %ebp
pushq %rbx
movl %edi, %ebx
subq $24, %rsp
call Q // <-- 1st call to Q (may clobber rdi, rsi, rdx, rcx, r8 and r9)
// prepare rdi, rsi, rdx, rcx, r8 and r9 for the 2nd call to Q
movl %ebx, %r9d
movl %r15d, %r8d
movl %r14d, %ecx
movl %r13d, %edx
movl %r12d, %esi
movl %ebp, %edi
movl %eax, 12(%rsp)
call Q // <-- 2nd call to Q
addl 12(%rsp), %eax
addq $24, %rsp
popq %rbx
popq %rbp
popq %r12
popq %r13
popq %r14
popq %r15
ret
The call to Q is allowed to clobber the registers rdi, rsi, rdx, rcx, r8 and r9 (among others). These registers contain the arguments of the call to P (i.e., a, b, c, d, e and f, respectively) and their values are still needed for the second call to Q. For this reason, the registers rbx, rbp, r12, r13, r14 and r15 are used to copy those registers before the call to Q in order preserve the original arguments P was called with.
Q is not allowed to clobber these registers that are used to save the callee-saved registers, since these are caller-saved registers, i.e., their value has to be preserved between function calls (another term for them would be call-preserved registers). Therefore, the code in the function P saves these registers onto the stack with push before clobbering them and restores them with pop from the stack before the control flow has returned from P (P is the caller of Q here). The net effect that the caller of P will see, is that those registers have their original value, the same value they had just before the call to P.
Note that the second call to Q may clobber the registers rdi, rsi, rdx, rcx, r8 and r9 as well, but there is no interest in preserving their values, since they are no longer after Q returns.
For a deeper insight into register clobbering, have a look at What registers are preserved through a linux x86-64 function call.
