Backing up a bit to give design comments from Project Mercury:
The Mercury astronauts had taken part in the development of their spacecraft, and insisted that manual control, and a window, be elements of its design. As a result, spacecraft movement and other functions could be controlled three ways: remotely from the ground when passing over a ground station, automatically guided by onboard instruments, or manually by the astronaut, who could replace or override the two other methods. Experience validated the astronauts' insistence on manual controls. Without them, Gordon Cooper's manual reentry during the last flight would not have been possible
Furthermore, astronaut group 1 and group 2 both required test pilot experience, and group 3 allowed military fighter jet experience instead of test pilot experience. Group 4 brought in scientists and gave them pilot training if needed, so overall the astronaut pool for Gemini and Apollo was made up of people that were expected to be able to fly the craft, and considering the astronauts were leaning on the design of the craft to make it user-pilotable they expected to be doing the flying: Project Gemini
It became known as a "pilot's spacecraft" due to its assortment of jet fighter-like features, in no small part due to Gus Grissom's influence over the design...
Given the features included for manually piloting and the fact that the person flying the craft was always a pilot, I'd guess that the pilot doing all the flying was the method preferred by the people on the craft. I'd have sworn I'd read that somewhere too, but it's been a while and I'm not finding that reference.
That said, the Apollo unmanned test flights demonstrate that everything necessary could be controlled from the ground:
- Apollo 4: Launched unmanned on a Saturn V into a circular parking orbit, tested re-igniting the S-IVB to get to an elliptical orbit, separated the command module and used the service module engine to adjust the orbit twice. Landed 16 km from the target landing site.
- Apollo 5: Launched the lunar module unmanned aboard a Saturn IB. The computer aborted the planned maneuver after 4 seconds due to a miscommunication about launch configuration, so
The ground controllers moved to an alternate plan to fire the descent engine manually two more times. They then performed the "fire in the hole" test and another ascent engine burn.
- Apollo 6: Launched aboard a Saturn V but had performance issues and ended up in an elliptical orbit instead of the planned circular one. The S-IVB failed to restart so they used the service module to raise the orbit, which left them short on fuel for finishing out the planned tests. Landed 80 km from the planned touchdown point.
Furthermore, this answer discusses remotely controlling the lunar module to crash it into the moon on most lunar missions after transferring the astronauts back to the command module.