On my site, I have a few "special" pages who's only purpose is to ban bots that ignore robots.txt (or worse, use it as a hint for where "the good stuff" is!). Here's how I do that:
robots.txt is a file that can be used to tell bots and search engines what pages are permitted to them, and what is forbidden. Some bots, however, do not bother checking this file first. Worse than that, others have been known to specifically hit the Disallow pages, on the assumption that anything blocked must have good content! (Of course, this is false-- anything private will usually be secured by authentication, but since when has logic stopped a spammer?).
On my site, I have several hidden pages. In order to prevent well behaved bots from being blocked, they're all listed in robots.txt as
User-agent: *
Disallow: /path/to/pageI have a special page there that actually blocks anyone who accesses it. The first page is at /cgi-bin/guestbook.cgi, but has no hyperlinks pointing. This is to catch spam bots that are hard coded to look for a common location for an exploitable script. I also have a link on my main page (with no link text, so a user won't see it) to /site/S.P.A.M.T.R.A.P/. Since a user using a screen reader could potentially hit that page, there's a doorway page there that warns the user not to go any further. Beyond there are two more links to other files (symlinked to the original .cgi) which will block an IP.
So how does it actually work?
The page consists of a perl script that adds a line to the .htaccess file which says the following: SetEnvIf Remote_Addr ^<IP ADDRESS>$ denied for each blocked address.
I also have
<Files *>
order deny,allow
deny from env=denied
allow from env=allowed
</Files>In my .htaccess file, which tells it to actually use the environment variables!
The actual perl scripts looks as follows:
#!/usr/bin/perl
use Socket;
print "Content-type: text/html\n\n";
$sendmail = '/usr/sbin/sendmail -i -t';
$htaccess = "/home/mikeage/public_html/.htaccess";
$domain = "mikeage.net";
$warning_to = "ip_ban";
$warning_from = "ip_ban";
$date = scalar localtime(time);
$remote_agent = $ENV{'HTTP_USER_AGENT'};
$remote_addr = $ENV{'REMOTE_ADDR'};
$inetaddr = inet_aton("$remote_addr");
$remote_host = gethostbyaddr($inetaddr, AF_INET);
$remote_addr =~ s/\./\./gi;
$abuse1="abuse@" . $remote_addr;
$abuse2="abuse@" . $remote_host;
(-w $htaccess) or do {
print "Not writable!";
die $!;
};
open(HTACCESS,"+< $htaccess") || die $!;
flock(HTACCESS,2);
seek(HTACCESS,0,0);
@contents = <HTACCESS>;
unshift(@contents,"SetEnvIf Remote_Addr \^$remote_addr\$ denied \n\#$date $remote_agent\n");
seek(HTACCESS,0,0);
print HTACCESS @contents;
truncate(HTACCESS,tell(HTACCESS));
close(HTACCESS);
print <<__WARNING__;
<html>
<head>
<title>Die Spammer!</title>
</head>
<body>
<p>You have triggered a trip-wire. This script exists solely to catch people doing things
they shouldn't be.</p>
<p>As a result, your IP address ($remote_addr) has been blocked from this entire site. You will
no longer be able to browse $domain. In addition, I have been alerted to your presence and will
be reviewing the records for possible action with your service provider.</p>
<p>If you have stumbled here by accident, you can email me at ip_ban at this domain to unblock
yourself. Be sure to paste in the network address in parentheses above so that I can unblock you.
If you don't mail me, you will <strong>NOT</strong> be able to get back to this screen again -
you are <strong>BANNED</strong>.</p>
<p>However, should you feel like spamming some people, try the following email addresses:<br />
<a href="mailto:$abuse1">$abuse1</a> or <a href="mailto:$abuse2">$abuse2</a></p>
</body>
</html>
__WARNING__
open (MAIL, "|$sendmail");
print MAIL "To: $warning_to\@$domain\n";
print MAIL "From: $warning_from\@$domain\n";
print MAIL "Subject: \[Alert\] Bot Blocked\n\n";
print MAIL "IP address $remote_addr ($remote_host) has been blocked from accessing $domain
because it called $0 on $date. The agent was $remote_agent.\n\n";
close (MAIL);
exit;Note that I do return two email addresses: abuse@spammers.domain and abuse@spammers.ip. Maybe they'll wind up reporting themselves!
hi,
i installed your script and some bot gets trapped.
but, the result is my server showing error 500.
when i check the htaccess, the script create 3 line. first line is to deny the ip. second line showing the time it occured and started with #.
the problem is with the third line. it show the user agent but do not start with #
how do i fix this problem?
tia
dny
Hrmm… there should only be two lines, that look something like this:
Perhaps the user agent included an embedded newline?
The 500 is probably related, as a bad entry in .htaccess can cause a 500.
If you visit the page yourself, does it successfully block you?
i fix it by adding a # just before $remote_agent
btw. tnx for making this great yet simple script free.
i symlink formmail.cgi to this script and seems like it catch quite some bad bot
even that i never have any link to formmail.cgi anywhere.
Glad it's now working.
A lot of bots are hardcoded to look for a formmail.cgi ; in the old days, I also had mine symlinked to guestboot.cgi .
After months of using it, are you finding this technique successful?
Well, there's no way to remember how much spam isn't sent to me. I can tell you that I average about 10 sites a week blocked; usually that's one or two search bots (mostly from China) that ignore
robots.txt, and the rest appear to be spam harvesting PCs, probably virus infested home PCs based on the their IP addresses.We need help to reduce our spam. I had in mind using something like your "ixitan" symbols on our website, as it would certainly reduce the spam bots. How do we get it? What would it cost?
I'm reluctant to screen out all that appears to be spam as we wish to be able to communicate with folks in need — but that may be wishful thinking. We don't know a whole lot about what we need, but we know what we don't need. Any help?
Hi all!
As a fresh mikeage.net user i only wanted to say hi to everyone else who uses this board
HelloSeo News|Seo Software Review News on seo world and article free download tips by idnewsnet
http://www.clevelandct.com
Cleveland Corporate Transport is Cleveland's choice for personal and corporate transportation.
Is it convenient to talk at the moment? http://rahiraaiyj.de.tl pearl nymphet bbs how could you have tits that big and be so bad at giving a tit job? You're willing to take a dp and swallow the load, but you can't hold your tits together? How fucking hard is that?
I'm doing a phd in chemistry http://ajabysofi.de.tl bbs tgp pics i was able to watch a small piece of this yesterday, im back to see if i can catch the whole thing,,,i would absolutly LOVE TO WATCH MY WIFE SUCK A BLACK GUYS COCK but to also give everything else he wants from her,,and i mean anything
Could I make an appointment to see ? http://omaaayige.de.tl young nymphete pics Okay. When I watch a video like this a few things matter. One of them is, the guy has to have at least a decent looking cock (no homo). Sure it doesn't matter because I'm watching the lady. However, if I am grossed out by what she is fucking, it completely turns me off. IMO, uncircumcised cock is fucking disgusting. How women can be attracted to that, I do not know. Guys, get that disgusting fucking foreskin cut off. It serves no purpose and is very unsanitary if you ask me. This video was a win, until she pulled his dick out. Now it's a complete fail. I am told I have a sexy looking dick, so when I watch POV (which is supposed to make you feel like you are there) then I expect the pornstar to have one as well. Sorry for the rant guys.
I'd like to cancel a cheque http://afihygajaqam.de.tl images of nymphets bbs I've seen lots of videos of Nicole Ray and this is the first time i ever seen her gag giving a blowjob
I'd like to send this letter by http://iceqejeqoa.de.tl nude model mpeg Great video. I didn't even mind that you don't see that much penetration. I came off her reaction to getting fucked alone!
What sort of work do you do? http://ijujysoodeu.de.tl rumanian models girls This got my pussy as wet as hers!!! I love that Beautiful big blk dick and watching her sexy ass cum all over it!!! Totally hot vid
Good crew it's cool
http://dyefihaudy.de.tl petit models the only thing that surprised me was the ending. i rarely see phat ass girls do that. but she was uninterested in the dick it was pitiful jus a girl wit good looks and no actin skillz.i take that back but the screamin was real too
Could you send me an application form? http://mijyyoysu.de.tl little model thumbnails She should do a tutorial for the ladies out there. Great technique – Not too much slobber, enthusiastic, good tongue action, and swallows! She also does anal. If she can cook, she is my perfect woman….
It's serious http://cubasisylof.de.tl young nudist pussy OMG she is hot, she takes that huge cock in the ass and she swallow cum! more girls like that
My battery's about to run out http://barojaularun.de.tl loli nymphets underage I love gloryholes. i have yet to find a girl in one, but i still always get a good blow job.
What's the interest rate on this account? http://yyfugobyita.de.tl lolicon lesbian If anyone's heard of a Matthew Sharp in the porn industry or of any erotic media related to him, please see info section in my profile. Many thanks.
Another service? http://barojaularun.de.tl school young loli top Mandingo had the right idea licking that pussy like that. I would have stayed down there with my tongue in her pussy for a lot longer. That's where the pleasure is at. It makes the fuck that much better!
I've been made redundant http://yyfugobyita.de.tl new index lolit tommy has his ass played with a bunch. anyone have the clip from club ginger, him, sharon mitchell and ginger lynn. one of the hottest ever. please post
No, I'm not particularly sporty http://yhofisaqo.de.tl angels girls pedo Meh. She's too nervous. I'd rather watch someone like…well, anyone who can make you think they're enjoying it.