I cut bot traffic to my site by over 80% with a simple trap.

Last month, my hobby website OpenBenches.org transferred more than 2.5TB of data to visitors. As much as I'd like to think the site is enormously popular, it looks like it is just AI bots endlessly crawling the data and slurping up the images.

So I've built B⁶ to be an indiscriminate tool to eradicate these unwanted parasites.

It is very loosely inspired by Jeff Starr's Blackhole, but works even on sites with heavy caching.

Here's the high level description of how it works.

• Create a directory like /honeypot/ on your site
• Put some code in index.php within the directory
• Add the honeypot as an explicit "Disallow" in robots.txt
• Include a hidden link to to the honeypot on every page of your site
• If something accesses that page, write its IP address to your .htaccess
• The .htaccess bans the IP address from ever accessing your site again.

This blog post shows you how you can build this yourself and discusses some of the problems it might cause.

Bad Robots!

The file robots.txt lives at the root of your website. It contains a list of instructions to automated web crawlers. Respectful bots will read it and refuse to look at specific parts of your site.

Add these two lines to it:

User-agent: *
Disallow: /honeypot/

That says bots shouldn't visit any page in the honeypot directory.

Bots generally check for updates every 24 hours, so either wait a day or manually submit it to major crawlers.

.htaccess

The .htaccess file contains a list of instructions for how your server should process requests.

Find the bit of your .htaccess file which looks like this:

<IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteBase /

Edit it to look like this:

<IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteBase /

### BLOCKED IPs
    #   TEST-NET-1
    RewriteCond %{REMOTE_ADDR} "=192.0.2.1"
    RewriteRule ^ - [R=402,L]
### /BLOCKED IPs

It is important that these rules come before any other rewrite rules.

Change the end of this IfModule to look like:

</IfModule>

ErrorDocument 402 "Payment Required - Access Denied."

If an IP address is in the blocked IP list, the server will send an error code and a readable message. You can add whatever plain text you like here.

Wipe Them All Out

Inside the honeypot directory, create a file called index.php (or whatever you like). When this page is visited, it grabs the visitor's IP address and the name of its User Agent. It then places them in the BLOCKED IPs section.

//  What's their IP?
$IP = get_client_ip();

//  User Agent
$UA = isset( $_SERVER["HTTP_USER_AGENT"] ) ? $_SERVER["HTTP_USER_AGENT"] : "";

//  Get the existing .htaccess file.
$htaccess_file = "../.htaccess";
$htaccess = file_get_contents( $htaccess_file );

//  Split by newline.
$htaccess_lines = explode( separator: "\n", string: $htaccess );

//  Where's our start block?
$start = "###   BLOCKED IPs";
$end   = "###   /BLOCKED IPs";

//  Exit if the .htaccess doesn't have the block.
if ( !in_array( needle: $start, haystack: $htaccess_lines ) ) { die(); }

//  This is where to start from.
$start_block = array_search( needle: $start, haystack: $htaccess_lines );

//  Add the line after the start block.
//  Include a date.
$htaccess_lines[ $start_block ] = $start . "\n" . 
    "\t#\t" . date( "c" ) . "\t{$UA}\n" . 
    "\tRewriteCond %{REMOTE_ADDR} \"={$IP}\" [OR]";

//  Turn the array back to a string.
$newHtaccess = implode( separator: "\n", array: $htaccess_lines );

//  Save the new .htaccess file.
file_put_contents( filename: $htaccess_file, data: $newHtaccess );

function get_client_ip():string
{
    //  Adapted from https://github.com/ibericode/koko-analytics/blob/main/src/Resources/functions/collect.php

    //  X-Forwarded-For sometimes contains a comma-separated list of IP addresses
    //  @see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
    $ips = empty( $_SERVER["HTTP_X_FORWARDED_FOR"] ) ?
        [] :
        \array_map( "trim", \explode( ",", $_SERVER["HTTP_X_FORWARDED_FOR"] ) );

    //  Always add REMOTE_ADDR and HTTP_CLIENT_IP to list of ip addresses, if set
    foreach ( ["REMOTE_ADDR", "HTTP_CLIENT_IP"] as $key ) {
        if ( !empty( $_SERVER[$key] ) ) {
            $ips[] = $_SERVER[$key];
        }
    }

    //  Return first valid IP address from list
    foreach ( $ips as $ip ) {
        if ( \filter_var( $ip, FILTER_VALIDATE_IP, FILTER_FLAG_NO_PRIV_RANGE | FILTER_FLAG_NO_RES_RANGE ) ) {
            return $ip;
        }
    }

    return "";
}

Setting The Trap

Web crawlers look at a page, find every link on it, then visit those links. So we need to put a link to the honeypot on every page - but in such a way that normal users will not find it or click it.

This adds a link, removes it from the accessibility tree, and gives it impossible to click text (a zero width joiner).

<a id="honey" aria-hidden="true" href="/honeypot/index.php">&zwj;</a>

This CSS hides it from normal viewers and removes the mouse cursor from view.

#honey {
    display: none !important;
    cursor:  none !important;
}

Problems This Might Cause

It's unlikely that a human will find the link, click on it, and be banned. But if they do, consider put a message in the blocking text telling them how they can get unblocked.

Some useful bots ignore robots.txt. If they identify themselves via a User-Agent string (and they should) you can manually remove them from .htaccess. You could also code in an allow list exemption for them. But, of course, many bad bots lie about who they really are.

Conversely, not all bots crawl every link. This won't catch bots which respect robots.txt but are nevertheless problematic.

If you have thousands of different bots blocked, the performance of your server might slow down while .htaccess is parsed. There's weak anecdotal evidence that it adds a slight overhead. But, personally, I don't think that's worth worrying about.

Bots often use hijacked IP addresses. After the bot has moved, an innocent party might be blocked from your site. Each block has a timestamp associated with it. You could periodically unblock older entries.

Bots rotate through hundreds of different IP addresses. So the same bot might get blocked multiple times.

Bots might see the CSS marks the link as hidden and refuse to follow it. Instead of CSS, you could use invisible spaces or other characters which would be hard for a human to see but appear visible to a bot.

Similarly, a smart bot might refuse to visit a path called /honeypot/ so choose a better name for it. A smarter bot would see that touching the honeypot leads to a ban - so perhaps a selection of traps are needed?

As is implied in the name, this is a blunt banhammer. It doesn't give a warning, nor does it look at User-Agent, Country, or any other signal. You touch the honeypot and you receive an instaban. No warning and no automated way to appeal.

Alternatives

There are alternative ways to do this. Some bots don't send language headers, some use older versions of HTTP, they often have severely outdated User Agent strings. Those are all reasonable ways to block something which is probably a bot. There are likely to be a few false positives though.

But B⁶ has no false positives. A normal user can't follow the link to the trap and a well-behaved bot won't follow the link.

Things like Anubis require a lot of work to set up and annoy normal users. B⁶ doesn't require a specific hosting environment and is never shown to normal users.

It Works!

Since running B⁶ a week ago, I've blocked over 400 IP addresses and seen a significant fall in traffic. I've not had any complaints yet. Sadly, none of the bots have paid to remove the paywall!

At peak, my site was seeing 216GB of data transfer per day. That's now dropped to around 25GB.

I don't claim this is the most sophisticated bot blocker in existence - but I enjoyed building it myself. I'll probably add something to records the geolocation of the IP and maybe remove any block older than a certain age.

If you'd like to use it or improve this code, I'd be grateful if you treated it as AGPL v3 licensed.

fuckyeahchinesefashion:

OP: How an artist perceive color (cr叶无趣).

It’s been a long time since I paid any attention to Bill Dembski or Vox Day/Theodore Beale, but they seem to have found each other in an unholy marriage of stupidity. Vox has “published” a new article called MITTENS 3.0 — MITTENS is an acronym for Mathematical Impossibility of The Theory of Evolution by Natural Selection — and it attempts to use mathematics to disprove evolution, using the data from the Long-Term Evolution Experiment by Richard Lenski.

It’s not really “published” in the sense of scientific publishing. There was no peer review, but just a dump of his assertions to an archive with no critical assessment. He compares the data from the LTEE to known divergence rates between Homo sapiens and chimpanzees, asking whether documented mutation rates in E. coli are sufficient to account for the 35 million single-nucleotide variation between two primate species. He says not.

The numbers are clear. At the non-mutator rate, 191 fixations can be accomplished in 252,000 generations. The requirement is 205 million. The shortfall is 1,075,000-fold.

Unfortunately for him, he exposes his methodology. He looks at the rate for a single mutation to go to fixation, divides the number of generations (252,000) by that rate, and concludes that there has only been time to accumulate 191 mutations. That’s true, if each trait has to wait for the previous mutation to go to fixation before it can be selected/drift to fixation itself. This is not how evolution works. Evolution is a property of populations and involves massively parallel action on multiple traits and individuals at once. Vox is using the typical simple-minded sequential estimations that simple-minded creationists have been using for decades to disprove their cartoon version of evolution. It’s ridiculous.

But Bill Dembski sees a naive “disproof” of evolution and decides that he and the author must be bestest buddies, and grants him a long interview in which Vox gets to thump his chest and gloat. Nowhere in this interview does Bill bring up Vox Day’s long history of coarse racism and ugly misogyny, nor does he bother to dig into the mathematics of Vox’s “disproof,” which is surprising since Dembski is purportedly a mathematician. Instead, it’s a long exercise in sucking up to Vox’s ego.

For instance, Vox has an undergraduate degree in economics and East Asian studies from Bucknell University, a small private liberal arts college. He does not have a science degree, and by his own admission he did not exactly apply himself to his studies.

Bill: You graduated from Bucknell University with majors in economics and East Asian studies. Please describe your training in economics in detail. What mathematics, probability, statistics, econometrics, or modeling did the program teach you, and which parts of that training have proved most relevant to your work on evolution?

Vox: My econ program covered all of those things except probability, although I didn’t take a course in modeling because my focus was on the historical process of development in late-developing countries like Japan and the Soviet Union. I did independent studies under Michael Moohr and Robert Chernomas, so my studies were a little bit more Marxian than was customary at the time. I would say developing an intuitive grasp of statistics, their limits, and how important it is to understand them contextually is what proved to be most relevant to all of my subsequent work, in games, evolution, and philosophy.

That’s not a very detailed answer. Any student who tells me that their achievement was developing an intuitive grasp of a subject is bullshitting me. Then he confirms it: Vox Day was a slacker who learned nothing in college.

Bill: Did you pursue any graduate study, professional research, or sustained independent work in mathematical economics or mathematics more generally after Bucknell?

Vox: None whatsoever. My biggest educational regret is not dropping out after my first semester sophomore year. I consider academic credentials to be a pointless waste of time. At Bucknell I seldom went to class and sometimes didn’t even bother buying the textbook. Once my professors realized that I was doing fine academically without doing the reading or attending the classes, they switched me to independent studies. I barely went to to a class my senior year.

He wishes he dropped out after his first semester, he seldom went to class, and didn’t bother reading the textbook — and he probably thinks that means he must have been a genius, rather than a lazy goof. It doesn’t speak well of Bucknell or his professors that they decided to get rid of him by putting him in a program of independent studies, rather than flunking him. I sorta sympathize with them. You get attached to your students and want to help them succeed, so you coach them through and design a limited curriculum that matches their abilities to allow them to finish the program. I guess we’re doing the students no favor if that just means their ego is stoked.

So if he has no background in or comprehension of biology, how did he write MITTENS? He explains it all:

Vox: AI did all the calculations and wrote the first draft of everything. It also provided about 750 of the 20 effective criticisms that improved the paper. I don’t think most people understand the first thing about the proper way to use AI to augment your outputs. If you don’t really know what you’re doing, AI will do nothing but defend its training and convince you that you’re wrong. I address this in great detail in Hardcoded, a book I wrote about the process of writing PZ and TFG.

[PZ is his book, Probability Zero, and has nothing to do with me; TFG is The Frozen Gene, another book he hacked together with AI. Isn’t AI wonderful? So many words, so much nonsense.]

Apparently the proper way to use AI is to hammer its outputs through multiple iterations until it tells you the answer you want to hear.

Well, but surely he had the work reviewed and evaluated by his colleagues? Nope, he has no colleagues. He has more AI.

Vox: My first, foremost, and primary form of feedback is from my Red Team. For all my papers, non-fiction books, and translations, I run them through a Red Team Stress Test comprised of ChatGPT, Deepseek, Gemini, and Grok instructed to take a hard oppositional stance. They produce detailed and extensive criticisms, most of which are obviously erroneous in some way, but even when incorrect, often produce new lines of attack. I seldom bother with human feedback. But both PZ and TFG were sent out pre-release to about ten volunteer readers, who ranged from a world-class physicist to a number of mathematicians with PhDs. I got some very good suggestions for improvement from a few of them, and even an Introduction and an Appendix from Dr. Tipler. But none of them had anything to add that fundamentally altered the work; I discovered and resolved the Kimura and LTEE [Long-Term Evolution Experiment] anomalies on my own after the fact.

Oh, well, he’s got the approval of notorious crank Frank J. Tipler, who has written multiple books on the physics of the supernatural and also writes for the Discovery Institute. So much credibility, wow. I’ll have to quote Wikipedia on Tipler to give you a little background.

Frank Jennings Tipler (born February 1, 1947) is an American mathematical physicist and cosmologist, holding a joint appointment in the Departments of Mathematics and Physics at Tulane University.[2] Tipler has written books and papers on the Omega Point based on Pierre Teilhard de Chardin’s religious ideas, which he claims is a mechanism for the resurrection of the dead. He is also known for his theories on the Tipler cylinder time machine. His work has attracted criticism from many, for example cosmologist George Ellis, who has argued that his theories are largely pseudoscience.

The bulk of the wikipedia article is about all the people who reject his nonsense. But sure, Vox Day has one supporter with legitimate academic credentials, that he was too lazy/incompetent to earn for himself. Otherwise, he’s just smarter than Kimura and Lenski.

He really does think he has disproven all of evolution all by himself, with a paper that contains all of 11 citations (a number I’d find unacceptable in an undergrad student paper that was much more narrow in its scope). Even worse, 7 of those citations are to the LTEE, and contradict his conclusions, and 2 are to ape genome sequencing papers whose authors would be surprised to learn how their work is being distorted. At least he has 2 that reinforce his argument, sorta, but they’re both also written by Vox Day.

So this is an argument made by an unqualified buffoon who churned it out of a battery of AIs, written with negligible knowledge of the scientific literature, and rubber-stamped by a couple of biased dorks from the Discovery Institute. But have no fear, his anti-evolution sentiment is not based on religious dogma at all.

Bill: To what degree, if at all, do religious considerations motivate you, not so much in the arguments you make but in the fact that you are making arguments against neo-Darwinism at all? In 2008, you published a book titled The Irrational Atheists, taking on what were then “the New Atheists” — Dawkins, Dennett, Hitchens, and Harris. They all lean heavily on Darwin to support their atheism. Did your religious leanings or your anti-atheistic leanings at all motivate your work criticizing evolution? Briefly, can you state your religious faith, whatever it may be?

Vox: Not at all. My faith is not based upon the historical origins of homo sapiens sapiens or any other species. I believe in material evil and I believe that Jesus Christ is the only thing standing between Man and his enslavement by that evil. I believe in one God, the Father Almighty, Maker of Heaven and Earth, and of all things visible and invisible. And in one Lord Jesus Christ, the Son of God… In sum, I’m a Nicene Creed Christian.

And he has a flaming sword. Can’t argue with that.