Trustworthy of decimal place of converted DD from DMS?

Question

I want -122d 54m 8.41s W (DMS) to be converted as decimal (DD). I tried two different available online converters to do the calculation. Surprisingly outputs are different. It looks like they match up to 7th decimal places.

-122.902336120571
-122.90233611

Out of curiosity, I used open-source dms-conversion JavaScript library that handles conversion to test what value would be printed out. It came out as:

-122.90233611111111

Only common ground from this observation I can find is up to 7th decimal places matches. I am new to this subject area. I am not sure which decimal value I have to trust if I want it to be converted back as original DMS value.

I wonder each on-line converter and external JS library I tried out has different logic when it comes to truncate seconds?

If I need to implement conversion programmatically either using an external library or math formula available on-line such as this: Converting latitude and longitude to decimal values, how far I have to trust in decimal places for DD value?

The difference in location between -122.90233611 and -122.90233612 is about one millimeter. Does that difference matter to your application? — Dan C, Jul 11 '19 at 17:05
@DanC is absolutely right, more details in Measuring accuracy of latitude and longitude? — Taras, Jul 11 '19 at 17:12
@DanC, to be honest, I do not know b/c I am not sure how much deviation it would create by one millimeter range but linked article from Taras makes me feel comfortable with having precision up to 7th decimal places. Ty for your time both of you. — DaeYoung, Jul 11 '19 at 17:17

score 3 · Answer 1 · answered Jul 12 '19 at 08:51

3

Obligatory XKCD reference:

More detailed answer

answered Jul 12 '19 at 08:51

Ian Turton

81,417
6
84
185

Looks like it is an upcoming Geomanuscript – Taras Jul 12 '19 at 09:20

score 0 · Answer 2 · answered Jul 20 '23 at 16:28

In javascript you only have the data type float available. There isn't a true decimal representation of all numbers to the right of the decimal point. There are some javascript math libraries that will convert to string and back (to decimal) to get as close to "true" as you require for significant digits. It comes down to doing division with binary numbers. I won't list any libraries here because they change. I hope this will help you find your answer.

Trustworthy of decimal place of converted DD from DMS?

2 Answers2