In a nutshell, dynamic pressure. There are also other serious limitations, but this one is the hardest to overcome. Say, at 10 km altitude, atmospheric pressure is still a bit over one quarter that of atmospheric pressure at mean sea-level. A conventional launch vehicle will at that altitude only have reached velocity of approximately 850 km/h and stay subsonic. As it gradually increases speed and climbs out of the denser atmosphere, it will reach Max-Q (maximum dynamic pressure) at an altitude of about 40 kilometers at velocity of 900 m/s (3,240 km/h). Here's a Douglas T. Jackson's example for a Space Shuttle launch:

Variation in air density (ρ), velocity (V), altitude (h), and dynamic pressure (q) during a Space Shuttle launch. Source.
Now, let's be perfectly clear here, that other launch vehicles will have different ascent profiles, but all would have reached only a small fraction of orbital speed at an altitude of 10 km, which is probably quite a bit higher than you could bargain for as a release altitude of a rocket sled launch assist system. Orbital speed is roughly 7.6 km/s (that's per second!), and our Space Shuttle has reached at 10 km altitude only roughly 3% of that, but is already experiencing approx. 70% of its flight profile's maximum dynamic pressure.
So, if we want to release our launch vehicle from the ramp at much greater speed, we have a bit of problem here, and it goes like this:
$$q = \tfrac12\, \rho\, v^{2}$$
Where $q$ is dynamic pressure in pascals, $\rho$ is fluid density in kg/m3, and $v$ is fluid velocity in m/s.
You can see that it's quite similar to the formula for kinetic energy and just like it, changes exponentially with velocity $v$. So if we assume same $\rho$, and we want to only double our vehicles speed at release altitude, we have doubled release point's dynamic pressure it has to withstand. Or increased it 4.5 times if we want to triple release speed. Or made it 8 time worse by quadrupling release speed, and so on. Basically, for each $n$ increase in release speed, we have $n^2/2$ as much of pressure to withstand.
And with 5 times the speed at 10 km altitude and 12.5 times the dynamic pressure (nearly 9 times the Max-Q of Space Shuttle), we've barely reached a good one third of the orbital speed, still have quite some atmospheric and gravity drag to overcome, for which our released vehicle will require own propulsion. It will also experience more friction heating, acoustic vibration from ground effect and as it reaches Mach 1 if launched from a non-evacuated tube, or a strong barrier shock if launched from evacuated tube once it exits it.
It will also have to tolerate much stronger lateral forces than a vertically launched vehicle, so its construction will have to be much, much stronger. All of that will eventually make it heavier and have a much lower wet-to-dry mass ratio, and have greater difficulties reaching orbital speed from release point, whereon it is still a subject to the tyranny of the rocket equation. Conventional, vertically launched rockets are really so elegant when it comes to managing extreme power at small increments of stress on the vehicle and where it can tolerate them best. You forfeit any of that elegance once you shoot out as a bullet. It will go bang, and quite likely, your vehicle will go with it.
There are however few of these problems that remain if you constructed such launch ramp on bodies with no atmosphere, like for example on the Moon.