On my site, I have a few "special" pages who's only purpose is to ban bots that ignore robots.txt (or worse, use it as a hint for where "the good stuff" is!). Here's how I do that:
(more...)
On my site, I have a few "special" pages who's only purpose is to ban bots that ignore robots.txt (or worse, use it as a hint for where "the good stuff" is!). Here's how I do that:
(more...)
