Marketing & Business
piece written on the 31st August 2014 by  

It’s been quite a while since I wrote anything about SEO on this blog, mostly because I’m working on SEO almost every single day at my company.  However, on a recent project I was inspired by a tactic that not too many people know about despite having been around for a long time. Today I want to share this tactic with you, it’s easy to implement and it’ll keep a lot of your competitors away and potentially save you a lot of money.

The way Google decides where a website appears in the search results (ie. ranks) is based on a number of factors – between 200 and 250 would be quite precise. However, all of these different factors have different weights. What I mean is that some factors are far more powerful than others and one of those factors is acquiring links (the dirty phrase is link building) to your website. Although links are not as valuable as they used to be, they’re still highly weighted and every website that needs to rank must acquire them.

Secret Links

If you’re at all familiar with link building, you’ll know that one of the easiest strategies is to use some software such as Open Site Explorer, Majestic or Ahref’s and use the software to get a list of links that your competitor has. Let’s look at an example, a friend of mine runs a company called Mirror Edge Networks (Enjoy the link, Morgan) that specialises in Professional Audio Video Solutions. If I open up Google and search for Professional Audio Video Solutions, the top result belongs to a company called System Solutions ( The most obvious thing here is to open up Open Site Explorer (there is a free version) and extract a list of all the sites that they have acquired links from. Let me say you some time, click here to see it. You can quickly see that they’ve acquired links from Bizcommunity, Gearhouse and a few others. If you’re looking for links, you can now investigate those links and try to get them for your site as well, but this post isn’t about that.

So what is this post about?

It took me just a few minutes to find a competitor and extract a list of links they have to then go about acquiring those links for myself, but what if we look at this from another angle. What if we’ve spent a great deal of time and money building a link acquisition strategy and have acquired some fantastic links and a competitor comes along and goes through the process of acquiring the same links?! That’s a bit of a rubbish situation isn’t it? This post is about stopping your competitors from snooping your link building strategy.

Each of these tools I discussed, Open Site Explorer, Majestic, Ahref’s and so forth have robots that crawl a website when their tools are run. So, what if we blocked those robots from crawling our website, then our competitors couldn’t snoop on us, right? Right!

This can be done quite simply by using two files that more than likely exist (or they should!) on your website; the .htaccess file and the robots.txt file. Chances are you’ll have them already, you’re just possibly not maximising their full potential. Now, there are many ways to go about doing this, but below I’ve outlined some examples of the two files and what you can include in them to block these link snooping pieces of software. In the example below, I’ve used code to block the robot that Open Site Explorer uses (dotbot) as well as the robot that Ahrefs (AhrefsBot) uses.

Here is the code:

.htaccess file

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule ^.* – [F,L]

robots.txt file

User-agent: dotbot
Disallow: /
User-agent: AhrefsBot
Disallow: /

By including these snippets of code in your .htaccess and robots.txt files you’ll be preventing your competitors from seeing who is linking back to you. At this point I should mention that there are a number of different web services / software that is able to crawl your website so you’ll have to do some research. Majestic is another bit one and their bot name is MJ12bot for example.

It’s a really simple and easy way to make things that little bit harder for your competitors and to protect some hard graft that you’ve done on acquiring your great links!

Good luck :)