Is Defensio too aggressive? (8)
Defensio, the new startup launching an antispam system, has been in private beta test for bloggers for a month. They also have an API, which allow companies to use Defensio to filter requests and user interaction with their own applications and software. This is the case of Active Reload, which used it for Lighthouse, their new issue and bug tracking for web app development. Rick Olson (aka Techno Weenie) has just reported about their experience and for him, Defensio is on par with Akismet:
you can see that the performance of the two engines is similar, except Defensio is more eager to mark items as spam. I was expecting more missed spam from Akismet actually, because there hasn’t been a single spam that I’ve seen since we added Defensio support.
Here is my experience with Montreal Tech Watch:
I get to the same conclusions as Rick Olson, although I found Defensio too aggressive for my own taste. For instance, it marked one trackback inside Montreal Tech Watch blog posts as spam, or comments from local bloggers as spam, even though they didn’t post any link. Patrick Tanguay, who wanted to correct a recent post, was seen as a spammer, even though he is using Defensio too for his blog.
The problem here is that Defensio doesn’t put suspect comments in the “awaiting for moderation” section, so I have no way of knowing that someone left such comment, except than going through the spam folder every 6 hours or so. In this area, Akismet is more usable, as it will send you an email if it can’t decide if a comment or trackback is legitimate or not.
Now, I don’t know if Defensio just needs a little fine-tuning or this is inherent to the way their algorithm works. The good news is Defensio is learning, and there are less and less false positives each day. But I was wondering how users would manage if they had 10 times more traffic and comments than this blog. Don’t misunderstand me, I think Defensio is doing an incredible job against 2-year-old Akismet. I really would like them to succeed. But at the end, you need to get into a “standard” user’s shoes and tailor the product to them.
Thanks for the comments!
As you mentioned, Defensio “learns”. We provide a basic knowledge base for “out of the box” filtering but since this knowledge base is so generic, it won’t be perfect, nor is it meant to be. Defensio needs to see a decent amount of innocent and spam messages to differentiate one from another. As you noticed, it’s learning and improving with time. People who’ve been using D for a while get well over 99% accuracy. The amount of time it takes varies depending on your traffic. In the meantime, you have a nicely sorted quarantine to find mistakes; much easier than anything else on the market!
I believe that you misunderstand how Akismet works: It is not true to say that it will e-mail you if it is not sure whether it thinks something is spam or not. Akismet only return “spam or ham” (ie: no “I don’t know”). The reason you are getting an e-mail is because Akismet flagged the comment as ham, you told Wordpress to e-mail you everytime a new comment comes in and you configured WP to hold innocent comments with URLs in moderation instead of publishing them immediately.
You also mentioned that url-less comments were tagged as spam: although most spam comments contain links, not all do. Spammers recently started spamming without urls… don’t ask me why, but they do! :)
To make sure Defensio learns quickly, make sure that you disable ALL other spam filtering plugins (Defensio needs to see all the comments to perform best). I noticed that we don’t get a lot of comment traffic from MTW, so I suspect that you are blocking obvious spam before it even reaches Defensio.
We are continuously improving our code and we will soon roll out an update. We’re in beta and the amount of traffic we’re getting now is very valuable for us.
Hi Carl,
Ok i misunderstood Akismet - or to express it better, I forgot how it worked. With Akismet, I had many emails daily about false negatives, and I was happy with this system, because that would mean i wouldnt loose a comment. False negatives, in the other hand, are less manageable for me.
About the learning, i will update MTW about defensio in another couple of weeks and take into account your comments.
Akismet will bury false-positives (legitimates marked as spam) in a sea of spam and basically makes them unrecoverable unless you take the time to go through pages and pages of spam. We feel our interface is easier to use since false-positives will be at the very top of the quarantine (low spaminess value) if there’s an error.
Also, make sure that Defensio is the ONLY spam-filtering system you use, otherwise, some comments might not make it to our system leading to a longer learning period.
Looking forward to see how this evolves! ;-)
[...] Regardless, Defensio is taking a similar approach to spam blocking that Akismet is, that is using a large external database that learns what people flag as spam and what isn’t, the end goal being better spam protection for everyone. From Montreal Tech Watch comes this review: Now, I don?t know if Defensio just needs a little fine-tuning or this is inherent to the way their algorithm works. The good news is Defensio is learning, and there are less and less false positives each day. But I was wondering how users would manage if they had 10 times more traffic and comments than this blog. Don?t misunderstand me, I think Defensio is doing an incredible job against 2-year-old Akismet. I really would like them to succeed. But at the end, you need to get into a ?standard? user?s shoes and tailor the product to them. Source: Is Defensio too aggressive? | Montreal Tech Watch [...]
carl mercier: i agree with you with the false positives. back when I used Akismet, I had 2 problems like this when i got 2 emails saying that they couldn’t comment on MTW. this was a painful process, but the thing is false negatives are more frequent than false positives. can you make the system so that it sends emails notifications when a spammed object has been classified as spam but its spaminess level is below 30%? or at least put it into comments in moderation?
I have now deactived all spam filtering, except Defensio.
We’re working on something to ease that “pain”.
[...] has received good and bad reviews, but from what I understand, it really gets better with time as opposite to Akismet who [...]
[...] Regardless, Defensio is taking a similar approach to spam blocking that Akismet is, that is using a large external database that learns what people flag as spam and what isn’t, the end goal being better spam protection for everyone. From Montreal Tech Watch comes this review: Now, I don?t know if Defensio just needs a little fine-tuning or this is inherent to the way their algorithm works. The good news is Defensio is learning, and there are less and less false positives each day. But I was wondering how users would manage if they had 10 times more traffic and comments than this blog. Don?t misunderstand me, I think Defensio is doing an incredible job against 2-year-old Akismet. I really would like them to succeed. But at the end, you need to get into a ?standard? user?s shoes and tailor the product to them. Source: Is Defensio too aggressive? | Montreal Tech Watch [...]
Leave a Reply