SEMRush Employee Doubles Down on Bizarre, Misleading Link Data

I’d normally let this kind of thing slide, but my integrity has been impugned (Olga Andrienko has accused me of “yellow press type headlines and incorrect data” in her comment on the original post) so I feel obligated to respond. First, let me start by why I used the words “bizarre” and “misleading” and why I stand by them. To be clear, my original post said explicitly that I did not think SEMRush was falsifying data and “I do not think SEMRush is intentionally inflating their numbers“.

What makes SEMRush’s Data Bizarre?

In SEMRush’s IP reports, they will display all the IP addresses that link to you… [click image for larger picture]

However, when you click on the domain number (in this case [1]), you land on a page with an advanced filter which shows no domains associated with that IP.

This is Bizarre. You can’t have an IP address without a domain in a link index. You just can’t.

What makes SEMRush’s Data Misleading?

SEMRush entered Matthew Woodward’s Best Backlink Checker contest which determined the size of indexes based not on root linking domains but IPs and Subnets. Unlike Moz, Majestic, Ahrefs, Webmeup and, to my knowledge, any other link index on the web, SEMRush stores every IP address they encounter for a domain over a 6 month period and report them independently. Because sites change IP addresses or use multiple IP addresses as a load-balancing method, a crawler will sometimes encounter more IPs than referring domains. This can cause their index to show more IPs than referring domains.

Because of SEMRush’s different collection and counting method, they destroyed the competition – not just Moz. Importantly, if I had not discovered and corrected this difference, SEMRush would still be enjoying the false narrative that their index is the largest by a landslide, and SEMRush would never correct it. But there are two possibilities. Either…

1. SEMRush did not know their collection methods were different
2. SEMRush did know their collection methods were different

As an aside, I knew that Moz did not collect IPs or Subnets, so I wrote a conservative model to predict these numbers and fully disclosed to Matthew in advance as to the limitations of our data. I wrote to Matthew, which he was able to put in his original post, “The [my] calculations should, therefore, be taken with a grain of salt.” This is the kind of proactive disclosure we should all make during research.

Given the two options above, it seems to me that if SEMRush did know they collect the data differently (and given their responses where they claim their method is better), then they had a responsibility to explain that essential difference and how it would make an apples-to-oranges comparison between their index and their competitors.

If they didn’t know there collection methods were different, then while this was a mistake, it calls into question what other assumptions they have made about their data when comparing their index to others and when marketing their index as the biggest or the best.

Responding to Olga Andrienko

First, let me acknowledge that she was not speaking in an official capacity on behalf of SEMRush. Here opinions are in Italics and my responses immediately follow.

1. “The SEO tools industry had always been known for the respect among the members of our vast community”
   I agree, and I still think it is. This is why I twice indicated that I believed SEMRush’s data was not maliciously constructed.
2. “yellow press type headlines and incorrect data published about us”
   I believe I have defended the headline adequately above. Moreover, I published no incorrect data.
3. “You should have used the Referring IPs report”
   I used both the Referring Domains and the Referring IPs report. The first showed the discrepancy in counts, and the second showed you had IPs that were not associated with a root linking domain (ie: “Nothing Found”). Both reports were highlighted.
4. “each domain can have more than one unique IP address”
   Yes, and I made this abundantly clear in my article when I mentioned Round Robin DNS as an example.
5. “we have no discrepancy in our reports… Referring IPs report, we show all domains per unique IP.”
   Yes, you did and still do. In the Referring IPs report, I can click on an IP and it will show me no associated root linking domain (ie: “Nothing Found”). This is a discrepancy.
6. “It would be incorrect to download the info from the Referring Domains report and filter it”
   On the contrary, if you did this with any other tool, you would find the exact same number. I couldn’t rely on the referring IPs report because IPs were not and still are not adequately associated with domains in that report.
7. “we think the actual numbers are, in fact, a lot less misleading than prediction-based formulas and estimates”
   I agree. If I provide a model, I am clear to the data partner and, as Matthew’s post indicates, I told him to take the data with a grain of salt. The important thing is knowing your index and disclosing any idiosyncrasies.
8. “We are very transparent about what we show on the dashboards and have a lot of tooltips with explanations”
   If I had not discovered this difference in calculation, would SEMRush have proactively reached out to Matthew for a correction?
9. “so if you knew SEMrush as your employer’s competitor before”
   I never used SEMRush’s UI for links because, frankly, until recently it wasn’t a contender. And, honestly, I don’t use most sites’ UI/UX. I just interact with their API.

Concluding Thoughts

1. Don’t give data to a contest without proper disclosures.
2. If you can’t give a proper disclosure because you are ignorant of your data’s differences, that is your problem.
3. Don’t accuse me of “yellow headlines”.

And a Question to Olga Andrienko:

If I had not found this difference between the way SEMRush reports IPs and everyone else in the industry, would SEMRush ever have sent in a correction to Matthew?