Is Visibility Statute or Nautical? Decide for Yourself

by Haris

In a kayak, it's often good to know the distance to some destination or object. The human visual system is pretty bad at this task. I can easily recall many instances of paddling for hours without any apparent gain on large distant targets. It always feels that progress should be faster…

Kayak navigation books such as Burch or Ferrero provide some help. They give a simple formula for calculating distances at which objects with given elevations above the sea level will be seen. The range of visibility in miles is equal to the square root of elevation as measured in feet. For example, a light house 100 feet above the sea level will be visible from 10 miles away. The formula is for an observer at the sea level.

In a kayak, you are not at sea level. For an observer in a kayak cockpit, you can add 1.5 miles to the visibility range of an object as derived from the formula. One-and-a-half miles is the square root of the eye-level elevation above the surface of the water. On average, paddler's eyes are just over two feet above the surface. Here are some immediate applications—if you can see beach goers feet as they walk close to the water, you're roughly within 2 miles off-shore. That's 1.5 plus a little bit added for the elevation of the beach. Two kayaks will lose sight of each other when they are separated by three miles—one-and-a-half miles of visibility from each side.

This is where I got tripped up. The formula for the visibility range is supposed to give you statute, not nautical miles. Burch, then, suggests that the visibility range as determined this way is underestimated by about 15%. What this means is that 10-mile estimate for an object with 100' of elevation should actually be 11.5 miles instead. Funny, that just happens to be the distance in nautical miles. So what is going on here? Exactly how much off is the square root approximation?

Time to dust off high school trigonometry books. First, let's track down the mathematical solution to the problem. Then, we will have grounds to decide if the limits of visibility are statute or nautical and how precise they are.

Generalized shape of the Earth is basically a sphere. For our purposes, the sphere can be further reduced to a circle. An average radius from the center to the surface is approximately 3,440 nautical or 3,959 statute miles.

Any line that just touches the outside of the circle can be used to mark the range of visibility—objects whose height is above the line will be visible, the ones below will be obscured by the horizon. Another line can be drawn from the point where the visibility line touches the outside of the circle to the center of the Earth. This line will have two known properties: (1) its length will be Earth's radius and (2) its angle to the visibility line will be 90 degrees. The Earth's radius and the visibility line can be viewed as two sides of a right triangle. The hypotenuse of this triangle is equal to the Earth's radius plus the elevation of the object above the sea level. What we have here is a simple case of Pythagorean Theorem with two known sides of the right triangle. We can solve for the unknown third side. In our context, a² = c² - b² can be written as Visibility² = (Radius + Elevation)² – Radius².

Plugging in the numbers and plotting the results of this equation we get the following picture. If the result of the square root solution is read in nautical miles, then it underestimates the actual distance by 6%. In statute miles, the distance is underestimated by just over 18%.

There it is clear as day: mathematically, the square root of elevation (measured in feet) is much closer to geographical visibility expressed in nautical rather than statute miles.

So why, then, is the formula is presented as offering the solution in statute miles? I don't know the answer to this one…

Here are some thoughts. Mathematical solution gives the distance at which the top of the object will reach the line of visibility. So the question is if you be able to see the top as soon as it reaches the line? To me, it depends. If the object is a mountaintop, the answer is most likely "No!" The tip of the mountain will have to get above the line at least somewhat to be seen. A mountain that is 400 feet above the sea level will reach the line of visibility when the distance is 21.3 nautical miles or 24.5 statute miles. Square root formula tells you that, when you first see the tip of the mountain, you are 20 miles away. By the numbers, when you are actually 20 nautical miles away, the mountain is already 47 feet above the line of visibility. When you are 20 statute miles away, the mountain will extend by 133'. When will you actually see the mountaintop? I don't know but, personally, I would rather be closer than I think I am. To me, nautical mile interpretation is a more conservative one and seems to be more appropriate.

If you are paddling at night and it is a light that you're looking for, you will see the glow well before it reaches the theoretical line. The moment the actual light emerges from beneath the horizon will be determined more precisely than in the previous scenario. Again, if the height of the light is known, it's square root is more appropriately interpreted as the distance in nautical, not statute, miles.

What do you think?