Tag Archives: robots.txt

How-To Block Alexa Robot

It’s easy and can save you some bandwidth per month, just add the following to your robots.txt file: Additionally it saves you to get too much information concentrated on alexa.com. You can contact them to delete existing content as well. … Continue reading

Posted in Surviving the Internet | Tagged , , , | Leave a comment

Best Practice robots.txt

I did some research month ago and wrote a longer article about robots.txt. But for a quick lookup it’s too much to read, so today it’s time a for a simple best-practices check-list: Use ASCII encoding for robots.txt (see as … Continue reading

Posted in Pressed, Surviving the Internet | Tagged , , , , , | 2 Comments