Agena is a much smaller and less powerful stage than Centaur.
Astronautix gives Agena-D a loaded mass of about 6.8 tons, as compared with around 23 tons for Centaur (though this varies quite a bit with version changes). Centaur uses the RL10 hydrogen-oxygen engine, with specific impulse up in the 440-450 s range depending on version, but Agena used UDMH and nitric acid with a specific impulse of around 300.
I ran across a 1968 Bellcomm memo comparing possible unmanned lunar payload delivery systems; from a Titan IIID, Centaur could put 75% more mass into a moon-bound trajectory than Agena.
In contrast, Centaurs were used with the Space Shuttle
There was a plan to launch Centaur from the shuttle but it was abandoned after the loss of Challenger. Instead, solid rocket stages (PAM, IUS, etc) were used to deploy high-energy spacecraft off the shuttle. Solid rockets were considered to be much safer for carriage in the shuttle payload bay, and probably considerably cheaper.
What made the Agena family less successful than the Centaur family?
Depending on your metrics, it wasn't less successful. 365 Agena stages were flown from 1959 to 1987. As of 2019, 251 Centaurs had flown.