Why is it that the rsp register is not decremented in a leaf function as it is at the beginning of any other functions?

Question

Here's some information about my system:

• Ubuntu 22.04.3 running on Intel x86_64;
• ggc version 11.4.0;

I've noticed that the rsp register is not decremented in leaf functions as it is in non leaf functions.

For example consider this C program in a file test.c :

int fx(){
     
     int x = 30;
     int y = 34;
     int z = 45;
     return 30;  
}

int main(){    
    
    int a = 10;
    int b = 20;
    int c = fx();
     
}

After compiling it with "gcc test.c -fno-stack-protector -o test" I run the "objdump -dw -M suffix test" and I get :

0000000000001129 <fx>:
    1129:   f3 0f 1e fa             endbr64 
    112d:   55                      pushq  %rbp
    112e:   48 89 e5                movq   %rsp,%rbp
    1131:   c7 45 fc 1e 00 00 00    movl   $0x1e,-0x4(%rbp)
    1138:   c7 45 f8 22 00 00 00    movl   $0x22,-0x8(%rbp)
    113f:   c7 45 f4 2d 00 00 00    movl   $0x2d,-0xc(%rbp)
    1146:   b8 1e 00 00 00          movl   $0x1e,%eax
    114b:   5d                      popq   %rbp
    114c:   c3                      retq   

000000000000114d <main>:
    114d:   f3 0f 1e fa             endbr64 
    1151:   55                      pushq  %rbp
    1152:   48 89 e5                movq   %rsp,%rbp
    1155:   48 83 ec 10             subq   $0x10,%rsp
    1159:   c7 45 fc 0a 00 00 00    movl   $0xa,-0x4(%rbp)
    1160:   c7 45 f8 14 00 00 00    movl   $0x14,-0x8(%rbp)
    1167:   b8 00 00 00 00          movl   $0x0,%eax
    116c:   e8 b8 ff ff ff          callq  1129 <fx>
    1171:   89 45 f4                movl   %eax,-0xc(%rbp)
    1174:   b8 00 00 00 00          movl   $0x0,%eax
    1179:   c9                      leaveq 
    117a:   c3                      retq  

As you can see, in main we have subq $0x10,%rsp but in fx this doesn't happen.

My questions:

1. Does it have something to do with the System V ABI that gcc follows or it is just an optimization performed by the compiler?

2. What is the reason that warrants this behavior?

3. Is there a way to tell the compiler I do not want this behavior in leaf functions?

Answer 1

As you do not pass any parameters via registers and you do not call any other functions reserving the stack space is pointless.

void nn(int x);

int fx(void){     
     volatile int x = 30;

     nn(x);
     return 30;  
}

https://godbolt.org/z/7qE4nbdcn

Is it ok to compile a C program in an Ubuntu x86_64 system with the flag "-mno-red-zone" like "gcc file.c -fno-stack-protector -mno-red-zone -o file" ? Is this option gonna break something or cause some incompatibilities or it is just an optimization ?

It is safe. Basically, it is pointless in the userspace code. Is it an optimization? - yes sort of. Reserving the space is simply pointless unless your stack is larger than the red zone.

int fx(void){     
     volatile int x[100] = {30,};

     return 30;  
}

It will reserve the stack minus size of the red zone.

https://godbolt.org/z/8qb191x9E

I've read that the System V ABI "mandates" the red zone. Does this mean that this red zone DOES EXIST even if we use the -mno-red-zone option ? I mean, if we use this option we do not use the red zone but it still exists ?

It is not related to your code. The signal and interrupts handlers will move the stack pointer away from your one. To change it you will need to recompile the kernel and amend some of its code.