Perri Nelson's Website 

Bad Behavior?

I am not a spambot, but Bad Behavior thinks I am. I recently implemented trackback pinging on my site. That's when I ran into problems with Bad Behavior.

The Trackback API is fairly straightforward. It's a RESTful protocol. You simply send an HTTP POST to the trackback URL with a few fields and get an XML response. Writing a software package to do this is trivial.

Or so I thought.

This website is more than just a blog. It's also a playground for me to write my own ASP.NET applications. It didn't start out as a blog, it started out as a racing track and event listing service. Recently though I decided to turn it into a blog, and it has been fun. So much fun that the blog will continue as long as I can keep it up.

I still like writing software though. When I decided to turn the site into a blog I looked at using some of the blogging platforms that are available for that purpose. I downloaded the Wordpress package and the Moveable Type package. I even looked at using DasBlog, an open source blogging package written in C#. Ultimately I decided that none of those packages were what I wanted.

A blogging platform is nothing more than a simple content management and rendering system. It's not that hard to implement one. The thing is, there are so many features that people put into blogs or have come to expect from their blogging platforms. I want to support all of those features.

One of the things about writing software, at least for me, is the fun of re-inventing the wheel. But that takes a lot of time, and there's a lot of people that have already done it. It's certainly not something a developer can get away with all the time. After all, in the real world of software development we have to work under a tightly constrained schedule, and integrate with lots of other products.

So for many features I've had to rely on third party offerings. One third party feature that I hooked up to the blog early on was Haloscan's comments and trackbacks. Haloscan provides a cool service, and it's free.

Remember though, re-inventing the wheel is part of the fun, so I always had it in mind to replace these third party pieces as I could. One of the first things I replaced was the trackback feature. Instead of using Haloscan to support trackbacks, I implemented my own trackback support. I could receive trackbacks, but I still had to use Haloscan to send them.

Almost from day one I started receiving trackback spam. It was frustrating, but a little gratifying too. Somebody had to be reading my site after all for the spammers to find it. The problem with spammers though is that they just don't stop spamming. The spammers all seemed to be hitting a single post, so I put in a trap for them.

ANYBODY that pinged that post automatically had their IP address banned from pinging trackbacks or comments. I didn't leave the trap in place long, I started working on a more reliable method of blocking the spammers. In the short time the trap was in place though it caught 417 IP addresses and banned them, all of them hitting the same post.

The new method of blocking the spammers I chose was straightforward. If you ping one of my posts, my trackback handler retrieves the page you claim to have linked to mine. It then scans the page to find a reference to the post's permalink. If no reference is found, the trackback is rejected immediately.

When I replaced Haloscan for comments as well as trackbacks, I started getting comment spam. The method of blocking trackback spam wouldn't work for comments, so I picked up another third party tool, Akismet.

Akismet isn't perfect, but it's pretty darned good. It blocks quite a few spam comments, but it does miss a few. It also supports trapping spam trackbacks too, so I plugged it into my trackback handler.

Relying on third party services can be a pain though. Sometimes the third party servers are under such a heavy load that they're unresponsive. Just ask anybody that uses Blogrolling.com's blogrolls. They're great when they're working, but when everybody uses them, the bandwidth requirements at the server can be pretty high.

So, I support my own comment mechansim. I handle trackbacks myself. Now all that's missing is the ability to send pings myself. Well, it was all that was missing, since I implemented it this past week. One of the reasons I implemented it is the whole re-inventing the wheel thing.

The other is that Haloscan's pinger can really be a pain sometimes. When it works it's a wonderful thing. When it doesn't it's a pain. There are a few sites supporting trackbacks that give Haloscan fits. When I ping one of those sites, the Haloscan pinger page dies and returns an HTTP 500.

The linkfests supported by the Bloggers Oasis have made this a more frequent problem for me, since I generally ping all of the day's open trackback posts. Haloscan only supports pinging 5 trackback URLs at a time, and when one of them causes the pinger page to die, it's a pain to figure out which one it was. So I implemented my own pinger.

Which brings us back to Bad Behavior, and a few other issues. My trackback pinger has had troubles with several sites, so I still have to use Haloscan as a backup. I thought it should be simple to send trackback pings. The API is straightforward after all.

Interestingly enough, one of the problems my trackback pinger has is with Haloscan itself. For some reason, if I'm trying to ping a post supported by Haloscan's trackbacks, Haloscan tells me I'm pinging to quickly and that I have to slow down. It doesn't matter if I haven't pinged for hours, I still get the same error response.

I don't know why that might be the case. I can always send the Haloscan sites pings using Haloscan's pinger. When I do they always work. Unfortunately that defeats the purpose of having written my own pinger.

On the bright side, I've found that the trackback handler implementations that give Haloscan fits seem to work just fine when I use my own pinger. But a lot of Wordpress sites barf on my pings almost immediately. They return an HTTP 417 error code with the text "Bad Behavior".

That's not very friendly. When I tried to figure out what might be happening, I found that a 417 error code isn't a "bad behavior" it's meant to be "Expectation failed". So I googled "bad behavior" and found the Bad Behavior plugin.

I looked at their website to see how Bad Behavior handles trackbacks. It doesn't analyze the content at all, but instead it analyzes the HTTP headers to look for spambot patterns. And, apparently my pinger looks like a spambot to Bad Behavior.

What fun. Bad Behavior looks for evidence that it's a human using a regular browser to post comments to a website. When it's a trackback, it's not usually a human or a browser doing the actual post, it's a service that's performed by the blogging platform itself, or by the posting software, or by a trackback ping provider like Haloscan. In other words, it's a bot doing the posting. So naturally, it's going to look like a bot.

Bad Behavior has exceptions for various services that post trackbacks, but it doesn't know anything about my pinger. So I've had to look at the way it scans the headers to reject my pings. I've made some adjustments, making sure to send a user-agent with the headers and an "accept" header. I don't know if that's enough to get past Bad Behavior though.

Anyway, I hate spam, as I'm sure just about anybody that isn't a spammer does. I hate it so much I've implemented my own anti-spam strategies. Now I find myself having to analyze and work around one just to send legitimate pings.

I don't know if my work-around will pass muster, but you can bet the spammers are working on their own ways to get past the anti-spam software we all use.

The spam wars just keep escalating.

UPDATE: Well, it seems that I still have work to do to get around the problems with Bad Behavior. At least I'm getting a different error from Haloscan now...

Problem saving. Please contact Haloscan

I'm not going to let this get me down though. I know I can fix this. It's only software after all.

UPDATE #2: I think I've fixed it. It turns out that the .NET framework ALWAYS sends an HTTP_EXPECT header with the value 100-Continue whenever you use HTTP 1.1. Bad Behavior ALWAYS rejects a POST containing the HTTP_EXPECT: 100-Continue header, regardless of whether you're using HTTP 1.1 or 1.0.

What's the fix? Use HTTP 1.0! The .NET framework does not send an HTTP_EXPECT: 100-Continue header when you use HTTP 1.0.

It seems strange that to fix the problem I have to use an older version of the protocol. I suspect that this is a bug in Bad Behavior.

Trackposted to Outside the Beltway, Is It Just Me?, The Random Yak, Adam's Blog, Big Dog's Weblog, basil's blog, Stuck On Stupid, Phastidio.net, Thought Alarm, 123 Beta, Rightwing Guy, Wake Up America, The HILL Chronicles, third world county, Faultline USA, Woman Honor Thyself, The Crazy Rants of Samantha Burns, stikNstein... has no mercy, The Uncooperative Blogger ®, The World According to Carl, Pirate's Cove, The Right Nation, The Pink Flamingo, Diggers Realm, Dumb Ox Daily News, and Right Voices, thanks to Linkfest Haven Deluxe.

Trackback URI for this post: http://perrinelson.com/track.aspx?postid=410
Permalink URI for this post: http://perrinelson.com/2007/1/20/410.aspx

123beta trackbacked with "Open Trackback Weekend #38"

Wow... I can't believe it's the weekend already! Please leave a post and I'll be sure to send a tb ping your way. Keep in mind that I have to manually send pings so it may take a little while...

Basil responded with:

I'm running Bad Behavior, and it seems to have let your ping through. I mean, there's a TrackBack. Unless, of course, you're using one of those other pingers to do it.

If you find that pings you send to me are caught by Bad Behavior, let me know.

Perri Nelson responded with:

Basil,
Yes, yours is one of the sites I have a problem pinging with my own pinger. When my pinger fails I still resort to the Haloscan pinger.

Carl responded with: