POV-Ray : Newsgroups : povray.general : Server failure Server Time
17 Apr 2021 07:36:40 EDT (-0400)
  Server failure (Message 1 to 10 of 76)  
Goto Latest 10 Messages Next 10 Messages >>>
From: Chris Cason
Subject: Server failure
Date: 14 Mar 2021 22:10:49
Message: <604ec229$1@news.povray.org>
Folks,

The povray.org server had a catastrophic hardware failure on March 9 and 
it was decided that the best course was to replace it completely.

In doing so I chose to do a full, fresh install of the OS rather than 
attempt to re-install the old one, and only import the data from our 
backup that we needed to get the various services back up.

As the failed system was 15 years old and had been through multiple 
upgrades and patches this meant quite a lot of work to replicate configs 
with the latest software versions and required re-compiling all the 
custom tools (e.g. the NNTP to Web gateway) we have build over the 
years. Hence it's taken longer than I would have liked to get it working 
again, but IMO it's worth the effort.

The NNTP server is the first to be brought back online and at the time I 
post this is the only service running and publicly accessible. Posts 
made here will eventually migrate to the web view once that's sorted and 
back up; in the meantime only NNTP users will see this message.

Before anyone asks - yes I did consider pointing the web sites 
(povray.org, irtc.org etc) to another server and in fact I did for a 
short time, with a simple placeholder page at the root of each site. I 
quickly noticed from the logs, though, that almost instantly I was 
getting lots of web crawler traffic (google, bing, etc) and that the 
server was returning "404 not found" to all accesses excepting the root 
placeholder page that I'd put in.

That wasn't desirable as it tells the crawlers that the pages that they 
are trying to refresh are no longer there, and they also see the simple 
root page as no longer having any links. I considered changing the 
config so it returned the placeholder page for all URL's, but that had 
the downside that it would replace any cached content with the 
placeholder, screwing up future searches.

I decided that rather than muck around with other solutions (e.g. return 
a temporary failure for all URL's) I'd be better off spending my time 
getting the new server configured, so I pointed the sites back how they 
were. I didn't know at that point that the server was toast and thought 
it might be only a day or two and haven't had time since then to 
re-visit it.

Finally, I'd like at this point to give a MASSIVE shout-out to Andrew at 
Netplex Internet in CT, who not only provided us with the replacement 
server (a HP DL360 G5 that they were no longer using) but also stayed up 
until 5am on Saturday getting it set up and configured to the point 
where I could remotely log in and start the setup.

-- Chris


Post a reply to this message

From: Chris Cason
Subject: Re: Server failure
Date: 14 Mar 2021 22:13:51
Message: <604ec2df$1@news.povray.org>
Forgot to mention ... at this point it appears we've only lost about one 
day of data (primarily posts to the newsgroups) as I recovered from a 
backup taken the previous night.

-- Chris


Post a reply to this message

From: Mr
Subject: Re: Server failure
Date: 15 Mar 2021 05:03:21
Message: <0552f79d71dfa9154a9ae48416e0310f4ed8a3cb.camel@hotmail.fr>
On lun., 2021-03-15 at 13:10 +1100, Chris Cason wrote:
> Folks,
> 
> The povray.org server had a catastrophic hardware failure on March 9
> and 
> it was decided that the best course was to replace it completely.
> 
> In doing so I chose to do a full, fresh install of the OS rather
> than 
> attempt to re-install the old one, and only import the data from our 
> backup that we needed to get the various services back up.
> 
> As the failed system was 15 years old and had been through multiple 
> upgrades and patches this meant quite a lot of work to replicate
> configs 
> with the latest software versions and required re-compiling all the 
> custom tools (e.g. the NNTP to Web gateway) we have build over the 
> years. Hence it's taken longer than I would have liked to get it
> working 
> again, but IMO it's worth the effort.
> 
> The NNTP server is the first to be brought back online and at the
> time I 
> post this is the only service running and publicly accessible. Posts 
> made here will eventually migrate to the web view once that's sorted
> and 
> back up; in the meantime only NNTP users will see this message.
> 
> Before anyone asks - yes I did consider pointing the web sites 
> (povray.org, irtc.org etc) to another server and in fact I did for a 
> short time, with a simple placeholder page at the root of each site.
> I 
> quickly noticed from the logs, though, that almost instantly I was 
> getting lots of web crawler traffic (google, bing, etc) and that the 
> server was returning "404 not found" to all accesses excepting the
> root 
> placeholder page that I'd put in.
> 
> That wasn't desirable as it tells the crawlers that the pages that
> they 
> are trying to refresh are no longer there, and they also see the
> simple 
> root page as no longer having any links. I considered changing the 
> config so it returned the placeholder page for all URL's, but that
> had 
> the downside that it would replace any cached content with the 
> placeholder, screwing up future searches.
> 
> I decided that rather than muck around with other solutions (e.g.
> return 
> a temporary failure for all URL's) I'd be better off spending my
> time 
> getting the new server configured, so I pointed the sites back how
> they 
> were. I didn't know at that point that the server was toast and
> thought 
> it might be only a day or two and haven't had time since then to 
> re-visit it.
> 
> Finally, I'd like at this point to give a MASSIVE shout-out to Andrew
> at 
> Netplex Internet in CT, who not only provided us with the
> replacement 
> server (a HP DL360 G5 that they were no longer using) but also stayed
> up 
> until 5am on Saturday getting it set up and configured to the point 
> where I could remotely log in and start the setup.
> 
> -- Chris

Thanks a lot for your dedication. 
Many people are following and encouraging your efforts from the two POV
related Facebook groups:
https://www.facebook.com/groups/739767156410999
https://www.facebook.com/groups/fansofpovray


Post a reply to this message

From: William F Pokorny
Subject: Re: Server failure
Date: 15 Mar 2021 08:56:18
Message: <604f5972$1@news.povray.org>
On 3/14/21 10:10 PM, Chris Cason wrote:
> Folks,
> 
> The povray.org server had a catastrophic hardware failure on March 9 and 
> it was decided that the best course was to replace it completely.
> 
...
> 
> Finally, I'd like at this point to give a MASSIVE shout-out to Andrew at 
> Netplex Internet in CT, who not only provided us with the replacement 
> server (a HP DL360 G5 that they were no longer using) but also stayed up 
> until 5am on Saturday getting it set up and configured to the point 
> where I could remotely log in and start the setup.
> 
> -- Chris

Thank you Chris!

Thank you Andrew!

Bill P.


Post a reply to this message

From: Chris Cason
Subject: Re: Server failure
Date: 15 Mar 2021 10:02:08
Message: <604f68e0@news.povray.org>
Folks,

www.povray.org is back up but without the database. hof.povray.org is 
back as it didn't need a database. other sites (e.g. irtc) aren't back 
yet, I will get to them next.

If you feel inclined please have a poke around the site to see if you 
can see any breakages that aren't related to the DB not being up or 
related sites being down (e.g. news.povray.org won't connect). Other 
than that it should work. If you see any completely blank pages (as 
opposed to connect failures) on the site then that's likely a PHP error; 
please LMK.

It will take some time to audit all the PHP code for side-effects as 
we've gone from PHP5 to PHP7 (plus all the database calls have to be 
changed as the mysql driver was dropped in favor of mysqli in PHP7).

Also please note the mail server isn't back up, so any mails to 
povray.org will not go through. (Technically it is up and working, but 
the spam filtering isn't back in place and I don't dare open it up to 
the world until it is as I'd be swamped).

-- Chris


Post a reply to this message

From: Ash Holsenback
Subject: Re: Server failure
Date: 15 Mar 2021 10:24:49
Message: <604f6e31$1@news.povray.org>
On 3/15/21 10:00 AM, Chris Cason wrote:
> It will take some time to audit all the PHP code for side-effects as 
> we've gone from PHP5 to PHP7 (plus all the database calls have to be 
> changed as the mysql driver was dropped in favor of mysqli in PHP7).

i knew there would be some pain there... i suppose that's why i'm not 
getting any response from wiki yet.


Post a reply to this message

From: Chris Cason
Subject: Re: Server failure
Date: 15 Mar 2021 11:12:50
Message: <604f7972@news.povray.org>
On 16/03/2021 01:24, Ash Holsenback wrote:
> On 3/15/21 10:00 AM, Chris Cason wrote:
>> It will take some time to audit all the PHP code for side-effects as 
>> we've gone from PHP5 to PHP7 (plus all the database calls have to be 
>> changed as the mysql driver was dropped in favor of mysqli in PHP7).
> 
> i knew there would be some pain there... i suppose that's why i'm not 
> getting any response from wiki yet.

That's a whole different kettle of fish, unfortunately. The wiki, texlib 
and a few other things run in FreeBSD jails (an OS/process abstraction) 
that have their own completely separate set of files (basically the 
entire operating system). While I technically could try running them 
as-is (iffy as we've gone from 32 to 64-bitOS) it's certain that there 
would be a bit of mucking around. If I decide to upgrade them to the 
latest tools then all the PHP code in there has to be audited as well.

Realistically, for the wiki, unless it's possible to force it to run 
with the old php and apache binaries, we'd have no choice but to upgrade 
the entire mediawiki codebase to something current as it's way to big to 
manually audit/change each place the database is used.

-- Chris


Post a reply to this message

From: Ash Holsenback
Subject: Re: Server failure
Date: 15 Mar 2021 13:04:15
Message: <604f938f$1@news.povray.org>
On 3/15/21 11:11 AM, Chris Cason wrote:
> On 16/03/2021 01:24, Ash Holsenback wrote:
>> On 3/15/21 10:00 AM, Chris Cason wrote:
>>> It will take some time to audit all the PHP code for side-effects as 
>>> we've gone from PHP5 to PHP7 (plus all the database calls have to be 
>>> changed as the mysql driver was dropped in favor of mysqli in PHP7).
>>
>> i knew there would be some pain there... i suppose that's why i'm not 
>> getting any response from wiki yet.
> 
> That's a whole different kettle of fish, unfortunately. The wiki, texlib 
> and a few other things run in FreeBSD jails (an OS/process abstraction) 
> that have their own completely separate set of files (basically the 
> entire operating system). While I technically could try running them 
> as-is (iffy as we've gone from 32 to 64-bitOS) it's certain that there 
> would be a bit of mucking around. If I decide to upgrade them to the 
> latest tools then all the PHP code in there has to be audited as well.

last time i touched wikidocgen i was running php7 on my machine so i 
have /some/ confidence, however i've not kept up with mediawiki changes 
so i don't know what's in store for that... i'm most fearful of table 
name changes. didn't that bite us in the ass last time we upgraded?

we /do/ have a most recent sql dump correct? i haven't ran a mediawiki 
on my machine in a while. do you think there's some value in seeing if i 
can get that running to see if we can even get it to import?

> Realistically, for the wiki, unless it's possible to force it to run 
> with the old php and apache binaries, we'd have no choice but to upgrade 
> the entire mediawiki codebase to something current as it's way to big to 
> manually audit/change each place the database is used.

i would be surprised if the former... i /think/ we both know where this 
is going


Post a reply to this message

From: Chris Cason
Subject: Re: Server failure
Date: 15 Mar 2021 18:33:42
Message: <604fe0c6$1@news.povray.org>
On 16/03/2021 04:04, Ash Holsenback wrote:
> On 3/15/21 11:11 AM, Chris Cason wrote:
> last time i touched wikidocgen i was running php7 on my machine so i 
> have /some/ confidence, however i've not kept up with mediawiki changes 
> so i don't know what's in store for that... i'm most fearful of table 
> name changes. didn't that bite us in the ass last time we upgraded?

Your memory is likely better than mine, so perhaps there were. I hope 
there'd be some sort of upgrade script if so.

> we /do/ have a most recent sql dump correct?

We have a backup of the DB (not a dump specifically, just the physical 
DB directory). While I have yet to bring up mysql I don't anticipate 
problems.

> on my machine in a while. do you think there's some value in seeing if i 
> can get that running to see if we can even get it to import?

Yes, that would be great. I'll get the DB back running and do a dump of 
all the tables once the machine has been moved back to the data center 
(hopefully shortly). At the moment it's sitting in an office and has 
limited outbound bandwidth.

-- Chris


Post a reply to this message

From: Ash Holsenback
Subject: Re: Server failure
Date: 16 Mar 2021 10:57:32
Message: <6050c75c$1@news.povray.org>
On 3/15/21 6:32 PM, Chris Cason wrote:
> On 16/03/2021 04:04, Ash Holsenback wrote:
>> On 3/15/21 11:11 AM, Chris Cason wrote:
>> last time i touched wikidocgen i was running php7 on my machine so i 
>> have /some/ confidence, however i've not kept up with mediawiki 
>> changes so i don't know what's in store for that... i'm most fearful 
>> of table name changes. didn't that bite us in the ass last time we 
>> upgraded?
> 
> Your memory is likely better than mine, so perhaps there were. I hope 
> there'd be some sort of upgrade script if so.
> 
>> we /do/ have a most recent sql dump correct?
> 
> We have a backup of the DB (not a dump specifically, just the physical 
> DB directory). While I have yet to bring up mysql I don't anticipate 
> problems.
> 
>> on my machine in a while. do you think there's some value in seeing if 
>> i can get that running to see if we can even get it to import?
> 
> Yes, that would be great. I'll get the DB back running and do a dump of 
> all the tables once the machine has been moved back to the data center 
> (hopefully shortly). At the moment it's sitting in an office and has 
> limited outbound bandwidth.
> 
> -- Chris

ok i did some poking around... first off i'm 2 minor rev's behind on my 
opsys and i need to keep it there for the time being, so i think 
mediawiki v1.33 is the best i can do (v1.35 is the current supported)... 
mainly because i'm at php7.2.5

https://en.wikipedia.org/wiki/MediaWiki_version_history

worth continuing?


Post a reply to this message

Goto Latest 10 Messages Next 10 Messages >>>

Copyright 2003-2021 Persistence of Vision Raytracer Pty. Ltd.