To give you an idea of how insane the (presumably AI) scraper was hitting us, here's a
screenshot of the DNS analytics for the period in question (I can't use HTTP analytics
for the "before" period because at the time news.povray.org was not proxied by
Cloudflare and thus they have no data on it; the DNS queries are a substitute).

Keep in mind each DNS query for the IP address of news.povray.org does not equal one
connection, since the scrapers only need to query for the IP address once per client,
and it is then cached. This was demonstrated when I did turn on Cloudflare proxying
but the hammering continued due to the above caching, leading me to need to firewall
it off for a day or two.

We will never know how many individual IP addresses were involved in the attack but I
expect it to be in the millions. This is not unprecedented - we had to deal with a
network of scrapers hitting the f-lohmueller.de site a month or two ago. In that case,
while they were coming in fast, they weren't so fast as to overflow the listen queue
so we were able to log every request (it also helps that the FL site is fully static,
no PHP and no database, which is not the case for the news server). As JR can attest,
for that case alone the logs showed the requests coming from over 800,000 unique IP
addresses. Fortunately we were able to isolate them well enough to firewall those off.

In the DNS analytics screenshot (taken today) you can see that over the prior three
days there were about 29 million queries for the address of news.povray.org, averaging
130 queries/second. But that's over three days - you can see from the spike that
almost all were on Friday (AU time) and peaked at 2.6 million requests/hour (about 720
requests/second) just after 3pm, at which time I started tuning the Cloudflare ruleset
to keep bots out but allow normal visitors (there was a pattern in the HTTP requests).
I was trying to avoid having to use a challenge (the "are you human" dialog that web
visitors get) but in the end I had to turn that on, too, as none of the rules I set up
would bite hard enough. I probably could have done it with a regex, but that's only
available for Enterprise customers*

In the screenshot of the HTTP analytics - which only became relevant once I turned on
the proxy - you can see that in a single four-hour period there were over 17 million
requests, peaking at around 250,000 per minute at 3:15-ish, at which point I started
setting the bot rejection rules. These requests were only from clients that "saw" the
new IP address, as mentioned above there were an unknown (but very large) number of
bots still hitting the direct IP address, which Cloudflare doesn't see.

Overall this careless, stupid and needless deployment of a clearly broken scraper not
only took down the site for a while but also has cost me at least a day of lost
productivity, as well as requiring the DNS change which blocked NNTP access. It is
almost beyond belief that anyone would expect something like this to work. I can only
assume it's a so-called "vibe-coded" program and that the authors simply *did not
care* if it worked or not, they just deployed it anyway.

I have taken the time to document this not only to show our userbase why we had to
make the change, but also to make a public record of what's been going on behind the
scenes due to the race to train LLM's and the cheap availability of immense
residential proxy networks. I will also post this same data to povray.org's front
page.

Our site is not by any means the only one dealing with this sort of nonsense, and I
suspect it's going to get worse before it gets better.

* That said I am grateful to Cloudflare for providing us with a free "Pro" account
under their open-source initiative.

-- Chris

Post a reply to this message

Attachments:
Download 'dns-analytics-75%.png' (619 KB) Download 'http-analytics-75%.png' (131 KB)

Preview of image 'dns-analytics-75%.png'

Preview of image 'http-analytics-75%.png'