Here's a brief answer since some people are trying to close the question and prevent answers:
As @ikrase points out answers to the Physics SE question Why is the Peltier / Seebeck Effect's efficiency so low in practical devices? are helpful here.
Briefly, there are two main parts to an RTG's conversion efficiency
Thermodynamics limit
The fraction of the thermal power that can be theoretically converted to electrical power is called the Carnot efficiency. It is given by $\eta = (T_{hot} - T_{cold})/T_{hot}$. For example, the MHW-RTG has hot and cold temperatures of 1273 K and 573 K, with $\eta$ of 0.55.
Now why on a cold planet or in space is the "cold" side so hot? Radiation efficiency which scales as $T^4$. It's hard to radiate heat if you are not hot yourself!
Material properties limit
The thermoelectric materials limitations as discussed in depth in the answers to the linked Physics SE question are currently the biggest source of inefficiency.
The need to have low thermal conductivity and simultaneously high electrical conductivity; it needs to be a good thermal insulator and simultaneously a good electrical conductor. Semiconductors can fall into this general category, but unfortunately useful materials don't fall far enough into this category to be highly efficient.
A little extension from @Uwe:
Unfortunately thermal and electrical conductivity is related, both depend on the movement of free electrons. So aluminum and coper are good conductors for both electrical and thermal flow. If there is a material with good thermal isolation, the movement of free electrons must be very low, so this material could not be a good electrical conductor.
References:
Related here in Space SE: