1

I have the following MySQL function:

DELIMITER $$
DROP FUNCTION IF EXISTS `fn_vincenty_distance`$$
CREATE FUNCTION `fn_vincenty_distance` (
        lat1 FLOAT, lon1 FLOAT,
        lat2 FLOAT, lon2 FLOAT
     ) RETURNS FLOAT
    NO SQL
    DETERMINISTIC
    COMMENT 'Returns the distance in degrees on the
             Earth between two known points
             of latitude and longitude
             using the Vincenty formula
             from http://en.wikipedia.org/wiki/Great-circle_distance'
BEGIN
    RETURN  DEGREES(
    ATAN2(
      SQRT(
        POW(COS(RADIANS(lat2))*SIN(RADIANS(lon2-lon1)),2) +
        POW(COS(RADIANS(lat1))*SIN(RADIANS(lat2)) -
             (SIN(RADIANS(lat1))*COS(RADIANS(lat2)) *
              COS(RADIANS(lon2-lon1))) ,2)),
      SIN(RADIANS(lat1))*SIN(RADIANS(lat2)) +
      COS(RADIANS(lat1))*COS(RADIANS(lat2))*COS(RADIANS(lon2-lon1))));
END$$
DELIMITER ;

I cannot reproduce some of the airport distances in miles. I'm using 69.2 miles as the distance between each degree of latitude at the equator:

Case 1

SFO-DFW

fn_vincenty_distance(37.6188056, -122.3754167, 32.896800990, -97.038002010)*69.2

Returns 1463.70 which is close to 1464.373 from https://www.airmilescalculator.com/distance/sfo-to-dfw/

Case 2

MIA-ORD

fn_vincenty_distance(25.7953611, -80.2901158, 41.9745219, -87.9065972)*69.2

Returns 1200.77 which is not close to 1197.062 from https://www.airmilescalculator.com/distance/mia-to-ord/

Question

How can I get the same answer being produced from airmilescalculator.com?

This is to match the calculation by airlines, for example from AA.com:

enter image description here

Vince
  • 20,017
  • 15
  • 45
  • 64
Kermit
  • 111
  • 2
  • you need to work out your 69.2 miles for the correct latitude see https://gis.stackexchange.com/questions/142326/calculating-longitude-length-in-miles – Mapperz Jan 10 '20 at 18:04
  • Do "qualifying" miles have any fixed relationship to actual miles? – Ian Turton Jan 10 '20 at 18:19
  • I agree with Ian's statement. Your math aside, you're taking the 1197 miles from ORD to MIA as an authoritative fact from this random website. Without knowing how they calculated this value, I wouldn't compare it to real world math. – KHibma Jan 10 '20 at 19:50
  • You are using the spherical case of the Vicenty's formula , but the site seems to use the ellipsoidal method. Also, just multiply the ATAN2 output times the radius of the sphere in miles. – Gabriel De Luca Jan 10 '20 at 20:26
  • @Mapperz the referenced question calculates at a single point -- how would I do this for two points? – Kermit Jan 16 '20 at 02:16
  • @KHibma not trying to compare it to real-world math--I'm trying to compare it to airmilescalculator.com – Kermit Jan 16 '20 at 02:33
  • @Kermit you should use the haversine formula https://www.igismap.com/haversine-formula-calculate-geographic-distance-earth/ as your using mysql you can do this https://www.plumislandmedia.net/mysql/stored-function-haversine-distance-computation/ – Mapperz Jan 16 '20 at 02:43

0 Answers0