This question has a table that shows the dry weight of the LM grew by 1600 kg between Apollo 11 and 17.
What enabled this weight growth? I know some of it was enabled by improving descent engine performance (nozzle extension from Apollo 15 onwards), but I don't have a complete picture.
Did they reduce margins (when experience showed that those margins could be reduced)? Were there other improvements?
A little bit of this and a little bit of that. There wasn't much actual performance increase. The Saturn V had significant performance margins for lunar landing missions, and at the scale of that rocket, even small optimizations could pay for a 1600 kg payload increase. For example, the mass of the S-I/S-II interstage alone decreased by 1000 kg between A14 and A15 (though it regained some of that weight in the later missions).
From the Apollo 15 press kit (emphases mine):
The payload increases were achieved by revising some operational aspects of the Saturn V and through minor changes to vehicle hardware.
The major operational changes are an Earth parking orbit altitude of 90 nautical miles (rather than l00), and a launch azimuth range of 80 to 100 degrees (rather than 72 to 96). Other operational changes include slightly reduced propellant reserves and increased propellant loading for the first opportunity translunar injection (TLI). A significant portion of the payload increase is due to more favorable temperature and wind effects for a July launch versus one in January.
Most of the hardware changes have been made to the first (S-IC) stage. They include reducing the number of retro-rocket motors [from eight to four), re-orificing the F-1 engines, burning the outboard engines nearer to LOX depletion, and burning the center engine longer than before. Another change has been made in the propellant pressurization system of the second (S-II) stage.
The modifications to the F-1 engines increased thrust by only a fraction of a percent; the thrust variation engine-to-engine was larger than the average increase between the early and late engines.
The larger nozzle of the LM descent engine increased fuel efficiency by only around 1%. The fuel tanks were slightly extended but this was considered primarily as a way to extend the landing's hover time rather than to increase payload. The big operational change for the descent and landing was that the CSM performed the first part of the lunar descent maneuver, dropping the perilune of orbit on Apollo 15 from 109km to 18km before the LM undocked. This meant that the LM had a significantly shorter descent to fly, and the average amount of remaining descent propellant for the later missions was quite a bit more than the early missions.
Software! The book Sunburst and Luminary mentiones that there were also improvements to LM landing software. The early versions tried to use raw radar altitude to fly a particular landing profile. Later versions had a (crude) terrain model (very early TERCOM).
When flying over a mountain and without terrain knowledge the early software would use more fuel because it would have thought that the LM was descending too fast. Later versions would "know" to expect that apparent altitude change and fly over a mountain without needlessly burning fuel to slow down the (apparently too fast) descent.
From chapter "P66 AUTO":
There appears to be unanimous agreement that we should add the terrain model of the specific landing site we’re going to in place of the present “billiard ball” moon. This will eliminate some objectionable pitch excursions and will make the LPD work better. And more importantly, make it safe to land in interesting places even if that involved approaching over a mountain or a deep valley. The purpose of the “a priori terrain model” (as we called it) was to tell navigation about the shape of the terrain the LM was passing over. Consider landing on the far slope of a wide valley. As the valley drops away under the radar beam, navigation senses that altitude is increasing, or at least not decreasing fast enough, so the guidance equation closes the throttle and the LM drops faster. Then, as the terrain suddenly rises near the landing site, the spacecraft may find itself dangerously low. There was a similar problem if the LM had to pass over mountains.
