The algorithm is fairly simple. In a nutshell, it searches different possible platform elevations, computes the cut volume, dumps that into the fill area, and checks whether there is any left over or lacking.
In detail, the algorithm deals with a digital representation (TIN or grid) of an idealized surface which we can think of as the graph of an integrable function f of the location coordinates (x,y) having area measure dA. The cut/fill operation is performed within a defined region: a measurable set of coordinates, call it R. Any proposed platform elevation z determines a net cut-fill value equal to the integral over R of (f(x,y) - z)dA. This is a strictly increasing continuous function that diverges to +Infinity as z gets large and diverges to -Infinity as z gets very negative, so therefore it has a unique zero for some value of z.
Based on this analysis, all the software has to do is (a) be able to compute such integrals, which are volume estimates, and (b) find the zero of a continuous function of one variable (z). The former depends on the method of representing the surface. With a grid one would use a Riemann sum or possibly (for greater accuracy) a 2D generalization of the Trapezoidal Rule or Simpson's Rule. With a TIN, the surface is broken into triangular patches and the volumes are computed with standard formulas for prisms and pyramids. Root finding also is standard and simple to carry out: see Numerical Recipes for best practices and code. The Secant Method works just fine but Newton's Method might converge faster for many surfaces (an important consideration, because each function evaluation can require some effort).
