Folks,

While the final content of the v3.8 documentation is certain to change (at a minimum I
want to update the Windows section, plus replace all images with larger ones), we
thought it prudent to spin up a docset to test the generation process since it has not
been run since the server crash a while back.

As some of you may know, the documentation on the Wiki is considered the master, and
thus we have some PHP code which pulls the content directly from the Wiki database,
modifies it according to our needs (e.g. converting wiki markup to HTML, pulling the
images out of the wiki file storage), and then outputs the filesets I will refer to
below.

In addition a special version of the windows docs are generated (separate from the
tarfile) from which the HtmlHelp CHM file is generated.

Due to changes in our OS install post-crash (particularly the PHP version), the PHP
scripts had to be updated to work with the current setup.

I'd like to make a shout-out to forum regular jr who took on the task of getting this
all going and making the filesets I'm about to reference.

Because we have had immense trouble with AI bots scraping the site (they almost all
ignore any robots.txt directives*) and also because we do not want these pre-release
docs indexed by any search engine, I will not put the full path to the docs here; I am
sure you can work it out :)

They are on the povray site, in temp/05May2026Docs. There you will find three tarballs
(Mac, Windows and Unix) plus a Windows HtmlHelp file (.chm). There is also an online
viewable version of the unix docs (identical to what's in the tarball, I just
extracted it as-is).

I ask that you please do NOT share the full link anywhere public, and in particular
not in any site on the web that could be indexed by search engines.

* see my followup post as to just how bad it was

-- Chris

Post a reply to this message

Regarding the AI scrapers ...

Just in the last week we've been hit badly by them - its been going on for longer, but
it really ramped up in the past 7 days - one of which was stuck stupidly scraping
every possible permutation of the wiki, particularly the special pages, which require
much more DB time than ordinary pages. jr alerted me to the fact the system load was
sitting at 4 (normally it idles at around 0.3) and when I investigated I found the
wiki being absolutely hammered. I did a "tail -f" on the logfile and it was scrolling
too fast to easily read. The bot in question is IMO badly written as it's not as if
MediaWiki structure is uncommon (looking at you, meta ... 1.5m requests to the wiki
alone, consuming 76 GB of HTML).

Not to be outdone, I also found the old bugs.povray.org site being hammered, again by
a bot scraping every possible permutation of the content (sorting, filtering, etc ...
looking at you, apple).

Between just those two instances, just in the last week, they managed to generate more
than 30gb of log files alone. The total bandwidth consumed is not known as I did not
have cloudflare enabled for the bugs site (it is now), but I did for the wiki and,
well, I'll let the screenshots do the talking.

I have turned on cloudflare's AI bot defense which has quieted things quite a bit (see
the final image, aftermath.png - it was taken the day after I turned the bot
protection on). However I cannot yet enable cloudflare on the main website (I have to
fix a bug in the login process that only occurs when it's turned on) so it's still
vulnerable (as is the news server).

Look, see and understand why I don't want them darn AI bots finding something else to
chew on ...

Post a reply to this message

Attachments:
Download '76gb-75%.jpg' (168 KB) Download 'metrics1-75%.png' (504 KB) Download 'metrics2-75%.png' (418 KB) Download 'aftermath-75%.png' (412 KB)

Preview of image '76gb-75%.jpg'

Preview of image 'metrics1-75%.png'

Preview of image 'metrics2-75%.png'

Preview of image 'aftermath-75%.png'

On 15/05/2026 07:07, Chris Cason wrote:
> Folks,
> 
> While the final content of the v3.8 documentation is certain to change 
> (at a minimum I want to update the Windows section, plus replace all 
> images with larger ones), we thought it prudent to spin up a docset to 
> test the generation process since it has not been run since the server 
> crash a while back.
> ...

Many thanks, Chris, for your hard work !

Unix online documentation is very useful and the Windows version
looks fine, even with the present size of images.
--
YB

Post a reply to this message

On 2026-05-15 09:08, Chris Cason wrote:
> Regarding the AI scrapers ...

First at thank you to all working on POV-Ray!

Second, Chris, maybe put the AI part on the front page news. Many sites 
have been hit by this.

ingo

Post a reply to this message

Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> I'd like to make a shout-out to forum regular jr who took on the task of getting
this all going and making the filese
ts I'm about to reference.

Slightly disappointed that the shout-out to jr was not in ALL CAPS.  :D

Excellent to hear that things are getting spruced up a bit.

There are thousands of ridiculously large data centers that are being built, and
people are even losing their homes and farms to "eminent domain", water prices
and electricity prices are going up, and computer memory is commanding
top-dollar.

Data centers so large that economists have determined that the scale is far
beyond what any reasonable investor would build and expect to pay off with the
profit from expected consumer usage.

Something wicked this way comes.

If only Google would index our site as thoroughly as AI scrapes it . . .  :|

Keep up the good work.

- BE

Post a reply to this message