Optimising for Isp only is problematic, as it's simply:
$$I_{sp} = \frac{v_e}{g}$$
Which is the same as optimising for exhaust velocity.
With no constraints on thrust, particle accelerations can achieve velocities arbitrarily close to the speed of light (The LHC is 3 m/s close). That's an Isp 30.6 million seconds, which can't be directly used in the usual rocket equations since you will have to account for relativistic effects.
Worse yet, photon thrusters can achieve thrust without expanding mass at all, achieving a force of $F = \frac{P}{c}$ (3.3 Newtons per gigawatt). At that point, Isp as a useful measure is utterly broken.
These are both possible to build. Particle accelerators have been with us for decades, and photon thrusters ("lightbulbes") for two centuries.
Some minimum acceleration is therefore required
Tier 1, able to lift off from the ground.
Chemical propulsion is unrivalled for thrust, and top out at around 450-460s for LH2/LOX. Exotic Lithium-Hydrogen-Fluorine Tripropellant systems have been demonstrated up to 542s, but those are highly impractical.
Generalising thermal rockets, current materials can withstand temperatures enabling an Isp of around 1,500s. This is lower in practice, as the proposed power source is usually a reactor. RD-0410 demonstrated a specific impulse of 910s.
Tier 2, interplanetary speeds within years.
Ion thrusters have been experimentally tested up to 10,000s, and used in space up to 3,100s
Improving the first category requires materials standing more heat, as heated gasses is the only propellant dense enough to achieve enough thrust. Alternatively, confinement of the exhaust must be done with something other than solid materials.
Improving the second category requires propellants that can achieve higher velocities than ionized atoms. No realistic systems are capable of producing a great enough amount of subatomic particles to achieve enough thrust.