Measuring Link Index Freshness

Disclosure: I work for Moz. One of the hardest metrics to compare link indexes on is their percentage of “live links”. At first, it seems like a simple endeavor – just download the backlinks and check if they are live. It turns out that the process is far more complicated for a number of reasons. Redirect chains may not include the target domain in the URL, and there is no way to know which outlink begins the redirect chain without following every outlink on that linking page. Links aren’t always links – does the index count form action as a link? iframe? embed src? Linking sites might be temporarily down (do you count that as a missing link or ignore it). How do you know if it is temporary or permanent? Different indexes have...