This is most easily solve using the equidistant azimuthal projection.
Guess some point for D, e.g., A. Transform A, B, C, to equidistant
azimuthal projection using D as a center. In projected space, solve for
D' (i.e., D' lies on BC and is a distance s from A). Project D' back to
lat,lon and update D with this position. Repeat. This converges
quadratically.
To see this in action, grab
http://www.mathworks.com/matlabcentral/fileexchange/39108
and http://www.mathworks.com/matlabcentral/fileexchange/39366 from MathWorks File Exchange.
Write a .m file to solve the planar problem, planesolve.m. Here is a
not-very-elegant version
function D = planesolve(pts, s)
% pts is 3x2 matrix of [A;B;C] lying in a plane. Find point D which lies
% on B-C and which is a distance s from A. There may be 0, 1, or 2
% solutions. Here assume there are two and pick the one most in the
% direction B-C.
xa=pts(1,1); ya=pts(1,2);
xb=pts(2,1); yb=pts(2,2);
xc=pts(3,1); yc=pts(3,2);
% write D = t*B + (1-t)*C, solve for t
t = ((yc-ya)*(yc-yb)+(xc-xa)*(xc-xb) + ...
sqrt((-xb^2+2*xa*xb-xa^2+s^2)*yc^2+...
(((2*xb-2*xa)*xc-2*xa*xb+2*xa^2-2*s^2)*yb+...
((2*xa-2*xb)*xc+2*xb^2-2*xa*xb)*ya)*yc+...
(-xc^2+2*xa*xc-xa^2+s^2)*yb^2+...
(2*xc^2+(-2*xb-2*xa)*xc+2*xa*xb)*ya*yb+...
(-xc^2+2*xb*xc-xb^2)*ya^2+s^2*xc^2-2*s^2*xb*xc+s^2*xb^2))/...
((yc-yb)^2+(xc-xb)^2);
xd=t*xb+(1-t)*xc; yd=t*yb+(1-t)*yc;
D=[xd,yd];
end
Then run the following script (in either matlab or octave)
g=[41.12507719625815,-86.32899412316736;...
41.16845596588609,-86.29237110757369;...
41.16923568742915,-86.19689279712584];
s = 5*1760*36*0.0254;
gd=g(1,:);
for i=1:5,
[x,y] = eqdazim_fwd(gd(end,1),gd(end,2),g(:,1),g(:,2));
d = planesolve([x,y],s);
[lat,lon] = eqdazim_inv(gd(end,1),gd(end,2),d(1),d(2));
gd = [gd;lat,lon];
end
format long;
gd
This prints out
gd =
41.125077196258147 -86.328994123167362
41.167417462129485 -86.406778034888063
41.167417401850848 -86.406778092275701
41.167417401850948 -86.406778092275616
41.167417401850948 -86.406778092275601
41.167417401850948 -86.406778092275601
which gives the converging sequence of approximations (lat,lon) for D. (Here the WGS84 ellipsoid is assumed. eqdazi_fwd and eqdazi_inv take an optional argument which lets you specify the ellipsoid.)