This all started a few months ago when we noticed specific clients had a HUGE spike in Referral Traffic out of nowhere. Sometimes a client will acquire rented ad space without our knowledge, so we started looking closer at the Referral Sources in Google Analytics. This is when we noticed that the new Referral Sources appeared to be tools for website optimization or SEO:
Was it us? Was someone here using new tools I didn’t know about?
No.
It quickly became clear we are dealing with spam. Around here we call it bot traffic and it’s pretty easy to tell it’s fake, as generally it’s 100% bounce rate and the names of the referral sites always include something like “buttons”, “site”, “traffic” or even “SEO”. Stupid spammers.
Unfortunately, Google still displays this traffic in Google Analytics, which means that clients see a spike in traffic (overall and referral specifically) but it’s not backed by increased time on site, page views, phone calls, form fills or conversions.
This traffic is actually hurting, not helping – and skewing our reporting. So, now what?
You Should Exclude Spam Bots from Google Analytics
As mentioned above, if you are reporting to clients or even simply using Google Analytics to track and manage your marketing campaigns, you need accurate data. Who wants to send a client a report with hundreds or sometimes thousands of referral visitors and zero conversion data to back that up? Not me.
And, Block The Bots
Also, who knows what these spammers are doing and what the bots are collecting? Every spammer has a reason for spamming, right? I mean, it seems like a lot of work to create these web bots and run these scripts, so it’s got to serve them some purpose. What if they are looking for vulnerabilities in order to hack in? No thank you.
How To Block The Spammy Referral Traffic
The good news is there are a number of ways you can block the referral spam. We suggest you block known spammers in the .htaccess of your site code and also filter the spam out of your traffic data using advanced Google Analytics filters. Below is the lowdown.
Editing the .htaccess File for Known Spammers
This part can become pretty technical, so it’s important to have a developer perform a full backup of your website before implementing this change. If there is anything wrong with the code in this file of your site, it could cause your entire site to go down. So if you don’t know what you are doing, please get a pro help you. (and if you screw it up, don’t say I didn’t tell you so!)
What is the .htaccess file and how can it prevent the spammers from hitting the site?
This hidden file located in your server cPanel, allows you to alter visitors who are able to access your site from a server level. We’re going to block the visitors we don’t want (referral spam) by not allowing them to hit the server at all.
To access the file on your server, simply log in to your hosting account and go into the cPanel. Your .htaccess file will be in the File Manager area but may be hidden. Check with your hosting provider to see the necessary steps for accessing this file.
Once you’ve located the file, implement the code to block all of the referral spam from hitting your server.
# Block Russian Referrer Spam
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*ilovevitaly\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*ilovevitaly.\.ru/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*ilovevitaly\.org/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*ilovevitaly\.info/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*iloveitaly\.ru/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*econom\.co/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*savetubevideo\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*kambasoft\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*buttons\-for\-website\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*semalt\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*darodar\.com/ [NC]
RewriteRule ^(.*)$ – [F,L]
Unsure of all of the spammers altering your referral traffic? You can always go into your Google Analytics account to find the spammers. Simply export all of the referral traffic sources from the past 6 months. Otherwise, there are comprehensive list of current spam bots that you can refer to.
Blocking Spammy Crawlers with Google Analytics Filters
One way these spam crawlers are accessing the Google Analytics servers are through inaccurate hostnames. To ensure the spam crawlers aren’t hitting your server with these hostnames and causing inaccurate data, you must verify to Google Analytics what the valid hostnames really are.
1. Create an unfiltered view in Google Analytics first because you can’t go back to an unfiltered view after implementing filters.
2. Locate your valid hostnames by going in the Reporting section of your Google Analytics profile using a wide date range. Under Acquisition –> All Traffic –> All Channels –> Referrals, select the Secondary Dimension for Hostname. You can export this list and identify the valid hostnames, which may include the following: your domain, sub-domains, or redirected domains. Anything else would be an invalid hostname. So keep the ACCURATE hostnames in a list.
3. In the Admin tab of your Google Analytics under View (make sure you’re on the main site view now, not the unfiltered), click Filters.
4. Add a new filter to include the valid hostnames you found.
Note: This method will NOT remove all of the referrer spam. You must also add additional spam filters in Google Analytics to filter all of the spam sources since the spammers can alter their hostnames to be a valid hostname. See the steps below for adding additional spam filters to prevent as much referral spam as possible.
Blocking All Referral Spam Sites with Google Analytics Filters
Since simply validating the hostnames isn’t enough to block spammers, you will also need create additional filters to block each individual site at a domain level. You do not need to include sub-domains since the domain itself should be enough to filter the sub-domains as well.
1. Locate referral sites that are currently spamming your site. You can do this by searching the web for the most accurate list of referral spam (already linked above), or you can export a list of referral sources from your Google Analytics. To know if it’s spam, you will notice the visitor behavior is non-existent. There will be 100% bounce rates and little to no visitor duration.
2. Create your list of sites using guidelines for the Filter Pattern using Regular Expressions provided by Google. Note: For the referral spam list filter pattern, you can keep it simple. As long as there aren’t any special characters within the domain. Keep this at a maximum of 255 characters. See the example below:
3. Create a new filter in your main analytics view entitled Referral Spam Filter #1.
4. Filter Type: Custom – Exclude. Filter Field should be set on Campaign Source.
5. Save this filter. Repeat this step a few times until you have used all of the referrer spam sites you’re finding in your analytics (because of the character limit in Google). The end result should look like the example below:
Exclude Bots with the Bot-Filtering Setting in Google Analytics
This is an easy one, we promise: The last step to make sure you have excluded as many spam crawlers and as much referral site spam you possibly can is to check your Google Analytics settings to make sure the bot setting is checked. In the Admin panel, click on View Settings. Check the box to exclude all hits from known bots and spiders. Save these settings.
Closing Thoughts
The steps listed above are not an end-all solution, unfortunately. New spam referrer sites are going to show up even if you do everything you can to try and prevent them. We suggest monitoring the site referral spam by checking in every 1-3 months (give or take) to see if you notice any new spam sites showing up in your Referral Traffic. If you do, simply follow the instructions above. There is word Google is also working on a solution to this problem, so we’ll keep you posted when we learn more.
Do you need help monitoring your Google Analytics for accurate data and reporting? We’re experts at that stuff! Give us a call: 800-871-4130
