There are quite many details here, I'll try to summarize the main points here.
The filesystem block size is the minimum allocation unit that can be reserved at one time. So, if a filesystem has a block size of 4096, a file that is one byte in size, still takes 4096 bytes on the hard disk.
This is due to the fact that the filesystem has to know which part of the hard disk belongs to which file. If the block size was smaller, then the allocation table would be larger. If the block size was larger, then even more space would be wasted with small files.
There are also other strategies for allocating space to files, depending on the filesystem used. But this is the basic strategy that is most often used.
However, on the application level, the bs parameter is the block size of an individual write/read operation fio uses when performing the benchmark. Larger write / read operation sizes can provide better performance, because several filesystem level block writes can be combined.
You cannot force applications to use any specific block size. They write / read exactly the amount of data they want to at any time. The sizes of individual write / read operations vary from a few bytes up to megabytes, all depending on the application.
fio is an exception here, because it is a benchmark tool.
Remote filesystems operate on top of the remote server's filesystem, and therefore the actual filesystem on the remote server specifies the block size used on the hard disk.
The protocol used when accessing the remote server's filesystem also has some "block sizes", most often defined by TCP segment size, which is determined from MTU. MTU is most often 1500 bytes in the Internet. In some local environments MTU can be 9000 bytes. This needs support from all parts of the network (switches, routers and NICs) to work properly.
