Calculating the distance between two lat and long: Why your math is probably wrong

You're looking at a map. You see point A and point B. You think, "Hey, I'll just use the Pythagorean theorem." Stop right there. Unless you’re measuring the distance between two crumbs on a kitchen table, that high school geometry is going to fail you. The Earth isn't flat. It’s a squashed ball—a "prolate spheroid" if we’re being fancy—and that makes calculating the distance between two lat and long a nightmare of trigonometry and planetary physics.

Most people just want a number. They want to know how far it is from London to New York or how long that delivery truck will take to hit the warehouse. But "distance" is a slippery word. Are you talking about a straight line through the Earth’s crust? A curve over the ocean? A path that accounts for the fact that the Earth bulges at the equator like a middle-aged man after Thanksgiving? Honestly, the method you choose depends entirely on how much you care about being precise.

The Haversine Formula: The Old Reliable

Back in the day, before we had supercomputers in our pockets, sailors used the Haversine formula. It’s the "good enough" method for most navigation. It assumes the Earth is a perfect sphere. Spoilers: it isn't. But for most apps and casual projects, the error rate is less than 0.5%. That’s basically nothing if you’re just checking how far away the nearest Starbucks is.

The math looks scary. It involves sines, cosines, and square roots. Specifically, the formula is:

$$d = 2r \arcsin\left(\sqrt{\sin^2\left(\frac{\phi_2 - \phi_1}{2}\right) + \cos(\phi_1) \cos(\phi_2) \sin^2\left(\frac{\lambda_2 - \lambda_1}{2}\right)}\right)$$

Where $\phi$ is latitude, $\lambda$ is longitude, and $r$ is the Earth's radius (usually about 6,371 km).

If you try to code this, remember that computers love radians, but humans love degrees. You have to convert those coordinates first or your distance will be thousands of miles off. I've seen senior developers pull their hair out for hours just because they forgot to multiply by $\frac{\pi}{180}$. It happens to the best of us.

Why the Haversine fails at high precision

The Earth is fat. Because it spins, centrifugal force pushes the equator outward. This means the radius at the equator is about 21 kilometers longer than the radius at the poles. If you're calculating the distance between two lat and long points over a long haul—say, Tokyo to Paris—the Haversine formula starts to wobble.

It treats the world like a marble. But we live on a lumpy potato.

For flight paths or long-range ballistic trajectories (not that you're building those, hopefully), you need something meatier. That's where Thaddeus Vincenty comes in. In 1975, he published an iterative algorithm that accounts for the Earth being an ellipsoid. It’s incredibly accurate, usually within 0.5mm. Yes, millimeters.

But there’s a catch. Vincenty’s formula is "iterative." The computer has to run the numbers over and over until it settles on an answer. Sometimes, if the points are nearly antipodal (exactly on the opposite sides of the world), the formula fails to converge. It just gives up. You’re left with a "NaN" error and a headache.

✨ Don't miss: Nudes on Social Media: Why the Rules Keep Changing and How it Actually Works

The Great Circle vs. The Rhumb Line

Here is something that trips up almost everyone. If you look at a standard Mercator map—the kind hanging in every classroom—a straight line looks like the shortest path.

It’s a lie.

On a sphere, the shortest path is a Great Circle arc. If you fly from San Francisco to Tokyo, you don’t fly "west" in a straight line on the map. You curve way up toward Alaska. Pilots do this because it saves thousands of gallons of fuel.

Then there’s the Rhumb Line. This is a path with a constant bearing. If you point your compass North-East and never change direction, you’re following a Rhumb Line. On a map, it looks straight. On a globe, it’s a spiral that eventually wraps around the pole. Sailors loved Rhumb Lines because they didn't have to keep adjusting their steering. But it’s a longer way to travel.

When you calculate the distance between two lat and long, you’re almost always getting the Great Circle distance. Just don't be surprised when that "straight line" looks like a massive curve on Google Maps.

✨ Don't miss: Amazon SDE 2 Interview: Why Most Senior Engineers Actually Fail

Coding it: Don't reinvent the wheel

Unless you’re doing a PhD in Geodesy, do not write these formulas from scratch. You will miss a parenthesis. You will mess up the floating-point math.

• Python users: Use the geopy library. It has distance.distance() for Vincenty (or the modern Karney method) and distance.great_circle() for the quick stuff.
• JavaScript fans: Look at turf.js. It’s the gold standard for geospatial analysis in the browser.
• SQL/PostGIS: If you have a database full of coordinates, use ST_DistanceSphere or ST_DistanceSpheroid. The latter is slower but accounts for the Earth’s bulge.

Real-world example: A ride-sharing app. If the app uses Haversine to calculate your fare, it might undercharge or overcharge you by a few cents compared to a more precise model. Over millions of rides, that’s a lot of money. They usually use routing engines like OSRM or Google Directions which don't just use coordinates—they use actual road data. Because, you know, humans can't fly over buildings in a straight line.

What about the "Manhattan Distance"?

In cities, lat and long distance is kind of useless for navigation. If you're at 5th Ave and 34th St, and your destination is 6th Ave and 42nd St, you can't walk through the buildings. You have to walk the grid.

This is called the "Manhattan Distance" or L1 norm. You calculate the difference in latitudes and the difference in longitudes separately and add them together. It’s a crude way to estimate walking distance in a grid-based city. It’s surprisingly effective for quick-and-dirty logistics planning before you plug things into a heavy-duty API.

The weirdness of the Poles

Things get truly bizarre near the North and South poles. At the equator, one degree of longitude is about 111 kilometers. At the poles? It’s zero. Every line of longitude meets at a single point.

If you’re trying to find the distance between two lat and long coordinates near Antarctica, your standard formulas might freak out. This is why polar explorers and scientists use specialized coordinate systems like Universal Polar Stereographic (UPS). If your app is going to be used by researchers at the McMurdo Station, you’ve got a lot more work to do than just a simple Haversine calculation.

Common pitfalls to watch out for

I've spent years looking at GPS data. Most errors don't come from the math; they come from the data itself.

1. The "Null" Island: If your longitude and latitude are both 0.0, you aren't in a magical void. You’re in the Gulf of Guinea, off the coast of Africa. This usually means your GPS failed or your database has a bug.
2. Coordinate Swap: Half the time, people get (Lat, Long) mixed up with (Long, Lat). GeoJSON, the standard format for web maps, uses (Long, Lat). Google Maps uses (Lat, Long). If your distance calculation says it's 8,000 miles to the grocery store, you probably swapped your coordinates.
3. The Datum: Not all "lat and long" coordinates are the same. Most use WGS84 (what GPS uses). But some local maps use older systems like NAD27. If you mix them, your points will be off by hundreds of meters.

Putting it into practice

If you're building something today, start with the Haversine formula. It's fast, it's easy to understand, and it works for 99% of use cases. If you find that your distances are consistently slightly off—especially near the poles or over very long distances—switch to the Vincenty or Karney algorithms.

For those just trying to get a quick answer for a project or a trip, use an online calculator that specifies it uses the WGS84 ellipsoid. It’s the same model your phone uses to keep you from getting lost.

Actionable Next Steps

• Check your coordinate order: Verify if your source data is Latitude/Longitude or Longitude/Latitude before running any calculations.
• Audit your precision needs: If your project involves distances under 10km, Haversine is perfectly fine. If you’re tracking trans-continental flights, move to the Vincenty method.
• Use specialized libraries: Avoid manual math. In Python, install geopy; in Node.js, use turf.
• Account for altitude: Remember that these formulas calculate distance at "sea level." If one point is on top of Mount Everest and the other is in Death Valley, the actual distance is slightly longer than the 2D coordinate distance.

For most of us, the world is plenty big enough that these small mathematical discrepancies don't ruin our day. But knowing they exist? That’s what separates a hobbyist from a pro.