tl;dr it was very dangerous
I would love to quote a document that says "we compute there was a 57.4356% chance of successfully completing an RTLS abort"...but I'm pretty sure that document does not exist. The NASA human spaceflight program was incredibly resistant to doing probabilistic risk analyses on their missions at least for Apollo and Shuttle. They were finally forced to do it for Shuttle and the results were as gloomy as you might expect but even then it didn't really address aborts.
Nonetheless I feel justified in saying that RTLS would have been dangerous for two reasons:
Reason 1:
You did not even think about doing an RTLS unless you were already in a very dangerous situation - one where flying backwards through your own exhaust plume at high Mach numbers seemed like a good idea in comparison. There were two categories of these situations:
Vehicle Performance
Which means, one or more of your main engines has failed or become degraded to the point that you cannot make it to any kind of orbit. So if you don't do something, you are going to end up in the Atlantic Ocean in a vehicle that can't survive ditching and that, for most of the program, was equipped with a bail-out system that I would call a band-aid except I do not want to smear the good name of Band-Aids.
It also needs to be mentioned that a main engine failure could be a Very Bad Thing. Those engines were equipped with a controller that monitored various sensors on the engine and would try to shut the engine off if those sensors showed that failure was imminent. But - and you will just have to trust me on this, the numbers come from Rocketdyne and they was argued over endlessly in Ascent Flight Techniques meetings but I don't have those minutes - when the controller was triggered to shut the engine down due to those sensor readings there was a ~33% chance that it would blow up anyway; there was a ~33% chance that it would shut down safely although it would have blown up if it kept running; and there was a ~33% chance that it would have run OK for the full mission, but would shut down safely.
The only actual in-flight failure leading to shutdown of a main engine was the most benign possible kind: the sensors failed and said there was a problem when the engine was actually running perfectly fine.
So a failed main engine may have shut down nicely, or it may have wreaked all kinds of havoc in the aft compartment. Nonetheless you have to try the RTLS or end up in the Atlantic.
Systems Problems
The reasons for doing an RTLS for systems problems are listed in the Ascent Checklist and are at least as scary as main engine problems. Basically they boil down to problems where a) you are going to die in a very short time if you do not land so it behooves you to get to the ground as quickly as possibly and in this phase of flight that means RTLS or b) you have suffered a failure that means you cannot survive a normal or TAL entry and the heating on RTLS entries is somewhat more benign so you have a better chance in that regard anyway.
They were:
- You are about to lose all your hydraulic systems, which means you will lose control of the vehicle.
- There is a huge leak in the cabin.
- All of the stuff you use to generate electricity is leaking out, and there are no batteries. When the electricity goes away, you are done.
- All of your heat rejection capability is lost and everything in the Orbiter is going to overheat and fail.
- Two of your three main electrical buses are down which prevents you from closing big doors that cover holes in the belly heat shield and therefore you will die on a normal or TAL entry. Oh and by the way, this will also cause a main engine to shut down.
- The outer window pane of one or more of the front windows is broken and pieces are missing. It will not survive a normal or TAL entry and neither will you.
I hope I have convinced you that the reasons for doing an RTLS are very dangerous in and of themselves.
Reason 2:
RTLS was completely untested. It was a certified abort, but it was certified by analysis and wind-tunnel testing, and 1970s computer simulation.
Exactly the same way that nominal ascents and entries were certified and were equally untested for the first launch...and STS-1 had quite a few bad surprises in store:
- Ignition overpressure from the boosters was underpredicted and caused mechanical damage to propulsion components in the Orbiter
- The plume effects on ascent performance were predicted incorrectly resulting in a significant difference in the ascent trajectory
- The hypersonic pitch trim on entry was incorrectly predicted
Any of these three errors could have resulted in disaster for the first shuttle mission if they had been a bit wronger or the program had been less conservative. It would be ludicrous to claim that the predictions for the RTLS ascent and entry trajectories were any better.
The unknown errors in analysis and prediction of the RTLS trajectory made it very dangerous.
In conclusion, flying on the Shuttle was dangerous anyway (see that PRA I linked); flying an untested trajectory because you have already suffered a dangerous failure or failures would have been dangerous squared.
