Are those pesky search engine robots crawling your site too much? Does the sight of even one result from a “site:” command at the engines boil your blood? Are you tired of seeing visitor after visitor trickle into your website from referrers like “Google”, “Yahoo” and “MSN”. Well worry no more, because below, I’ve listed the top 10 ways to keep those spiders far, far away.
#10 – IP Delivery
Not just for spammers anymore – there are now dozens of legitimate uses for delivering content based on the IP address of the visitor. One of the best things you can do for spiders is to put California (and Washington) on your list of non-served states. Maybe it’s because you don’t ship your wacky weeds there, or maybe you’ve got issues with far left politics. Either way, keeping IPs from these geographies pointed to a junk page will ensure that pesky crawlers turn tail and run.
#9 – Duplicate Content
If you and ten dozen of your best friends all scrape from already existing content and run it through your whiz-bang text re-organizer, you’re sure to get crawled, right? Wrong! Thanks to advances in duplicate content detection, you can use this as a great way to go on the ban list.
#8 – Reject Inbound Links, Just Submit!
Why bother getting links to your site when you can just submit to the engines on a daily basis. This tactic is recommend by many of the top ranking sites for “SEO” and “SEM” so why not take their advice?
#7 – Buy Your First Thousand Links
When your first thousand links are part of a link manipulation scheme, you can really boost your chances of being ignored by Google. Yahoo! and MSN though may still give you the crawl (and even the rankings), so make sure to spam report yourself (or wait a few weeks, savvy competitors can help save you the trouble).
#6 – Use 100% Pure, Grade A Flash
Everyone’s favorite animations can help you solve your spider infestation, too. Try an all-Flash portal and watch your crawling activity shrivel up and disintegrate.
#5 – Text is for Losers
Images, rich media, video, and those beautiful midi files are what make the web great, so don’t mess around with text in HTML.
#4 – Update Once a Decade
Search engines love fresh content, so starve them of it. If you update your website before 2010, you’re probably on the wrong path.
#3 – Splash/Intro Pages
Everyone knows that splash pages are fun and user-friendly, but did you know they can help keep out spiders, too? It’s true – just try forcing a spider to click a link in a Flash file or an image map and watch the magical results.
#2 – No Content Without User Action
Love AJAX? Javascript? Then this one’s for you. Until a user clicks, hovers or draws a unicorn with their cursor, don’t show them anything (and make sure to keep the code in a separate file to be called by your scripts).
#1 – The Beauty of NoIndex
Just one handy tag in your robots.txt or meta robots tag and all your problems are solved. If you’re not wanting to do it yourself, just leave your server relatively unsecure and someone else can do it for you.
I hope it’s not too early in the morning for a strong dose of irony 🙂
