leaderboard left
leaderboard right
content topleft content topright

Comparing GPS Route Quality

Jun
4
2007

One of the most common types of feedback we get from GPS purchasers is “I can’t believe it sent me that way to get to XYZ”. Judging route quality is a ridiculously difficult task. There are so many variables that go into route selection, that it is nearly impossible to judge the quality of a route against anything scientific. In the end, the only thing you can compare a route against is your own expectations. Here is why it is nearly impossible to judge route quality and derive any meaningful data.

Let’s tackle this from the perspective of wanting to create a valid, unbiased assessment of routing quality. We want a repeatable method for calculating individual routes and comparing each route generated against other devices.

Selecting Routes to Test

The first thing we would need to do is to pick routes….. Lots of routes! If you put a GPS into my hands, any GPS from any brand, I can quickly find for you probably a dozen examples where the GPS didn’t pick the best route. Therefore to make an effective test we need to overcome any possible bias that could be introduced. So perhaps taking a random sample of 50 routes might be a big enough sample size.

Then we need to make sure we don’t introduce any regional bias. Some towns and cities are just better mapped than others, so we would want a sample from a large pool of areas. Since we decided we want to judge against about 50 routes then perhaps one route in each state will eliminate any regional bias.

Now here comes one problem… many GPS devices don’t allow you to specify the starting location of a route, you can only use your current location. So without physically taking each device to each of our 50 starting locations many devices can’t be tested.

The Impact of Mapping

Another consideration is the map vendor, typically Tele Atlas vs NAVTEQ. Since the vast majority of GPS devices use data from one of those two mapping companies, manufacturers who use the same mapping data will likely come up with similar routes, and make similar mistakes if the underlying data wasn’t as good as it could have been. To make an effective assessment of roue quality (based on the routing algorithm), each device would need to be running the same version of the same mapping data. But let’s even ignore the map vendor for a moment and focus just on the overall device/map combination to judge how effective each is.

So now let’s say we have created fifty random routes and we are ready to judge route quality. At this point in this article I’m going to switch from talking about a hypothetical test to talking about a real world routing example. I’ve come up with a route to test on each GPS device which starts at the nearest major airport and goes to a grocery store near my home.

Actually Testing a Route

The test should be as simple as going to the airport parking lot, turning on the GPS, and creating a route to that particular grocery store and looking at the route it picks. Then I can compare that route to the route I know as being the fastest, and see which devices pass the test. Wait a minute…. is the route I picked as the “correct” route really the fastest, or the best way to get there?

Again, I’m no longer talking in hypotheticals, this is an actual example. I might think that I know the best route, but what would other people think? So I put the question to the test. I asked several dozen people what route they think is best to get from the airport to that particular grocery store. Even to my own surprise, I received eight distinctly different answers from people. While I thought there was only one, or perhaps two sensible ways to get between those points, a few dozen people came up with eight different routes that they each thought were the best.

I didn’t tell people why I was asking for their route pick (I was asking for this test) and I asked that they not use any technology to “calculate” a route for them and rather go from memory as much as possible although they could look at a map to get street/route names.

Therefore it is my conclusion that it is exceedingly difficult to create any test whereby you compare routing quality of multiple devices against each other without introducing a bias and still getting meaningful data.

  1. You need a large sample size of routes to compare.
  2. You need routes in a wide sample of locations and lengths.
  3. You need to be able to set the starting location on the GPS device or physically take the device to each starting location.
  4. You need to separate discrepancies in mapping data versus the algorithm itself.

My Route is Best… No, Mine is!

All of those hurdles might be overcome with a effort, except for this last hurdle… the human factor. Humans often can’t agree on the fastest route between two points, so who am I to judge which routes picked really are the best? Again, you might be tempted to say “well I drive that route all of the time and I know it is better”. I thought that too in my test, yet I was very wrong as illustrated by the number of people who disagreed which was the “correct” route in my example.

Of course I’ve seen various GPS devices calculate some pretty crazy routes. But keep in mind GPS devices are currently designed primarily for getting you somewhere that you don’t know how to get there yourself. And while you might think you have the best route, there are probably people out there who disagree.

Therefore I don’t often comment about the overall route quality different devices generate. It is just way too easy to introduce any bias and produce data which is truly meaningful.

Remember this episode of Seinfeld?

Kramer: No no no, they’re talking the West Side Highway, at this time of day that’s insane. They’re heading straight into gridlock. Oh, those fools.
[...]
Kramer: Boy, look at that. Se that’s that fire I was listening to yesterday.
Jerry: Wow, the whole building burned down.
Kramer: They just don’t know what street to take. You remember that time I got us to Yankee Stadium in rush hour in fifteen minutes?
Jerry: Of course.
Kramer: It’s all up here, Jerry. All up here. It’s innate.
Jerry: The amazing thing is you never have any place to go.
[...]
Kramer: Hey, you wanna come down the fire station with me?
Jerry: Fire station?
Kramer: Yeah, I made a map of my shortcuts. I’m gonna rock their world!
[...]
Captain: Well, Mr. Kramer, your list of short cuts is most impressive.
Kramer: Yeah, and this is just the Upper West Side. Wait until I get to the Village, then you’re gonna see a magic show.
Captain: Mr. Kramer, just about every week some brash young hothead like yourself saunters in here talking about faster routes and snazzier colors for the trucks, well, fact is we feel things are fine the way they are.
[...]
Dispatcher: Attention Company 390, structure fire at Leapin’ Larry’s Appliance Warehouse.
Kramer: Leapin’ Larry’s? Hey, that’s uptown. You gotta take Amsterdam.
Captain: Stay out of this, Kramer.

The Results

I decided to illustrate the results based on how many people picked each different route. Then I’ll add in each different online mapping service and various different GPS devices compared in picking a route. So how did each device perform in this one test? 88% of people thought the Garmin, Lowrance, and TomTom devices failed. 86% of people thought the Harman Kardon GPS failed. And 63% of people disagreed with the route picked by Magellan, despite having the highest percentage of human picks. Again, the sample size from this test is 1, so the results are completely meaningless. They do however highlight how difficult it is for people to agree on the best route and should set some expectations on how often you might disagree with the route your GPS picks.

Route 1: 94.36 miles
Picked by 37% of people
Picked by Magellan GPS devices
Picked by Google Maps (API) and MapQuest

Route 2: 76.39 miles
Picked by 19% of people
Picked by a Harman Kardon GPS device
Picked by DeLorme Street Atlas USA

Route 3: 100.91 miles
Picked by 14% of people
Picked by no GPS devices
Picked by no online mapping services

Route 4: 77.71 miles
Picked by 12% of people
Picked by Garmin, Lowrance, and TomTom GPS devices
Picked by Google Maps (website), Yahoo Maps, and MSN Maps

Route 5: 121.17 miles
Picked by 9% of people
Picked by no GPS devices
Picked by no online mapping services

Route 6: 94.72 miles
Picked by 5% of people
Picked by no GPS devices
Picked by no online mapping services

Route 7: 78.53 miles
Picked by 3% of people
Picked by no GPS devices
Picked by no online mapping services

Route 8: 93.43 miles
Picked by 2% of people
Picked by no GPS devices
Picked by no online mapping services

21 Responses


  1. Genius. Thank you! For illustrating so well, how challenging it is to compare routing data between gps devices without appearing biased. There’s a reviewer out there who chooses just three routing examples in all his garmin vs tomtom (and magellan) devices declaring sweet victory to Garmin every time. I took issue with his impartial and misleading 720 review.

    will - November 10th, 2007
  2. Did any GPS device or online mapping service pick a route that was not picked by at least one of your “experts”?

    jim - January 10th, 2008
  3. Jim – I didn’t have any “experts”. That was the point. :) These were ordinary people picking routes. The full results of the test are listed above.

    Tim - January 11th, 2008
  4. So if I read your results right the HK unit had the shortest route, the garmin/lowrance/tom tom next, and the magellan had the longest. But who is to say which is the quickest on any given day. However, I prefer the route that gives me the best mileage considering cost. Do these gps companies plan on calculating gas cost effectiveness into their gps units, where you could enter your ave. mpg on hwy and city for your car and give you the money saving route? I guess now I’m really wishing……someday.

    Alan - March 14th, 2008
  5. I wouldn’t look at it like that Alan. While the HK picked a shorter mileage route, that route probably isn’t the fastest. The point I’m trying to illustrate is that many people couldn’t agree on the same route, so you won’t often agree with the route your GPS picks either. They will all get it wrong some of the time.

    Most of the devices today are configured purely for “fastest” from point A to point B. However there are some models that offer an economical route type which will favor a slightly longer trip if it saves a good amount of miles (and thus less gas). But not many devices offer that feature.

    Tim - March 14th, 2008
  6. The day that manufactures choose to use Artificial Intelligence to calculate best routes, that will be my dream come true.
    Considering factors like:
    Current traffic status.
    The current time/date (rush hour, weekends, holiday, speed limit, and statistical data that can be learned by the device by keeping a journal log and possibly a user community interchange/share of data.
    I heard that tomtom is coming with a new model april 2008 that might address some of these parameters in their calculations of optimal routes.
    just my 2 cents…

    Phenix - March 17th, 2008
  7. Interesting concepts, good set of assumptions and believable results. From my perspective, the main purpose of a GPS device is to get you from point A to point B – WITHOUT GETTING LOST! It matters not if it takes 5 miles or 10 minutes longer than the optimum route.

    Sal - June 15th, 2008
  8. A TomTom One I was trying definitely picked the wrong way from north of Houston to Austin. It appears to calculate fastest route using about 40 mph for non-interstate roads! It’s route was about 100 miles, 1 1/2 hours longer than any of several other routes, on a 3 hour trip. Not good…

    James - July 2nd, 2008
  9. Thankfully the estimated time isn’t always the actual time. I think you would likely find that the new IQ Routes technology on the newer TomTom devices clears up issues like that in many places.

    Tim - July 2nd, 2008
  10. Do you know of a gps that will allow you to block interstate use for routing? I prefer not to travel on interstates.

    Brenda - October 21st, 2008
  11. Brenda, most all of the current street type GPS devices will allow you to avoid highways.

    Tim - October 21st, 2008
  12. My husband has a first generation TomTom and is interested in the newer TomTom Go which supposedly learns your favorite routes. I can’t find that feature in the TomTom descriptions. Help?? Thanks!

    Sunny - November 21st, 2008
  13. Sunny, TomTom doesn’t have such a feature.

    Tim - November 21st, 2008
  14. Thanks Tim – is there any unit (from any manufacturer) that has such a feature? He doesn’t remember where he heard this. I have a feeling they were talking of “advanced lane” feature or something else….
    Thanks!

    Sunny - November 21st, 2008
  15. TomTom has a feature called Active Lane Guidance which has a similar sounding name, however it doesn’t remember any routes. The only GPS I know of that will record a route you drive is the Dash Express with their My Route functionality.

    Tim - November 21st, 2008
  16. Eureka! The routing problem is biased for wavelength of light. Fortunately the spectrum is linear. As a crude guess about what to do, the algorithms should be developed in different lighting conditions, say a red room and a blue room. The routing problem is unknown in evolution from a gps point of view; ants worked it out at hoc and using scent trails and many trips. The gps routing problem, from a linear grid and addresses, is a new problem in evolution. It is that is an actual challenge to the human species. Wavelength is important in electronics – indirectly, everywhere, including living species. The best example is in eyes, where vision in red light is much different and less hazy than vision in blue light. That’s why Garmin is lagging; the newer companies started with some of Garmin’s experience and include better routing at lower prices already.

    Michael Lewis - June 2nd, 2009
  17. Are there any GPS units that will allow you to pick your own route. On a recent trip I had to keep picking points for the gps to take me to then when I was getting close to that point I needed to recalculate it to the next point other wise it was always trying to get me to take a ferry to cross over the river and I was looking for a bridge. Before my trip it would have been nice for me to look at a map pick the route I wanted to use then be able to enter it into the GPS and have it guide us the way we wanted to go. Or it would be nice if it did give us a few more options in the choices of the routes it picked instead of fastest or shortest. Tom Tom 510

    Jake - August 2nd, 2009
  18. I found it necessary to pick a few intermediate points, but with some thought there need not be many. For instance attempting to map a route from Seattle to Chicago, the Venture HC tried to send me to Yakima, then south to just east of Portland where I would pick up the interstate. Setting Ellensburg as a waypoint caused the GPS to stay on I-90 which heads straight east from here and that solved most of the rest of the route.

    Michael - August 3rd, 2009
  19. I’m a school bus driver with a two hour route that I run each morning. If I fill in for another driver that’s another route, with over 150 routes in the district it would be nice to simply see my route on a GPS instead of having to look at a printout.
    Is there a gps that will give me a defined route and NOT change if I drive away from the path and return to it periodically? Could it even create where I stop to pick up the kids?

    Thanks,
    David

    David - August 23rd, 2009
  20. I purchased a Magellan Maestro 5310 because it has a large, bright screen and the text to voice is very accurate and easy to understand. When I pick a destination and preferences, e.g. Shortest, Fastest, etc, it frequently picks a route that I would like to modify slightly. Is the a way to insert a waypoint and have it recalculate the route like my old Cobra did?

    Henry - January 9th, 2010
  21. I want a GPS that has an option that will allow me to avoid high crime areas and another option to select major streets with faster speed limits.

    marty - August 2nd, 2010



content bottomleft content bottomright