The Soviets tried direct remote control in the Lunokhod program. The radio delay was found to be a huge obstacle. They used a driver plus a spotter and a commander. The driver had to continuously integrate looking ahead (predicting the future, with only low-resolution B&W images to guide him) with controlling the rover via a delay. This was exhausting work and could only be done in short stints.
This is shown in a Lunokhod documentary (Tank on the moon?), I'll see if I can find it.
This article has more detail:
Only one member of each crew would drive the rover. Behind him would sit the crew commander, who would oversee the driver’s handling of the rover. Joining them in the control room would be a navigator, a radio antenna operator, and the flight engineer, who would monitor the rover’s systems. Each crew would operate the rover for two hours; then the other crew would take control.
...
Latypov and Dovgan’s [the drivers] only guidance came from a monitor, which displayed images from Lunokhod’s two low-resolution television cameras. To any video game enthusiast it sounds simple—but this was nothing like a video game. The cameras did not send a continuous stream of images, but rather single frames, like a slide show, at intervals that varied from seven to 20 seconds. And because radio signals took three seconds to travel round trip between Earth and the moon, the driver didn’t see the results of his actions for many long moments. For this reason, if crew commanders Nikolai Yeremenko and Igor Fyodorov saw Lunokhod heading toward catastrophe, they could push a button to halt the rover.
Dovgan, now 66, was well prepared by intensive training. “Driving on the moon felt even easier than it was in the lunodrome,” he says, but his comment belies the difficulties of navigating the rover. The low resolution of the slide show made it difficult to spot craters and boulders, especially at high sun angles, and there was a “dead zone”—a three-foot-wide area immediately in front of the rover that Lunokhod’s cameras could not see. The only solution, according to Dogvan, was to memorize the features in this area from the previous image, before the rover reached it. “When we were looking ahead and thinking of the obstacles that we did see, we also had to remember what was just behind,” he says.
(emphasis mine)
Also, like the Apollo crews, they found it difficult to estimate distances due to the lack of landmarks that have a known size.
Now, some of these problems can be alleviated with modern technology. You can get better imaging than 1 frame/10 seconds. You can improve the view around the vehicle. You can add computer analysis, and maybe integration of ground-based images with overhead map data to get a better idea of distances, slopes etc. But the fundamental problems of delay and having to interpret alien terrain remain.
