Intuition tells me that a bigger lens will collect more light with the same f-number than a small lens. But photographers tell me I am wrong. I cannot understand why.
There are quite a few posts both on SE (e.g. here and here) and elsewhere addressing this question. But I fail to understand it and would appreciate if someone could answer it slightly differently (ideally without photographic terminology, but from the purely optical perspective). For simplicity, let's say that we are trying to image an object located at infinity. Its image will be a single on-axis pixel on our camera sensor.
All rays passing through the lens will end up on the same pixel. The amount of light landing on this pixel will be proportional to the entrance pupil area and will not depend on the focal length of the lens. Thus, we can change the f-number (which is the f/D ratio) without changing the amount of light on the sensor. Why do they say then that the amount of light is the same for all lenses with the same f-number (for example, here)?
(I know about the T-stop number, but let's ignore the transmission for the moment. I am interested in geometrical treatment only.)

