On 3/22/21 7:22 AM, William F Pokorny wrote:
> On 3/21/21 5:04 AM, Chris Cason wrote:
>> Current status: www.povray.org is now back in read-write mode with the 
>> database hooked back in.
>>
> ...
> 
> Do I need to login to get at the forums / news.povray.org/groups from 
> the web page? This link just hangs for me currently.
> 

Read only access worked just now for me - please ignore my question.

Bill P.

Post a reply to this message

On 22.03.2021 18:05, Cousin Ricky wrote:
 > One thing that was wonky for a long time but seems to be fixed now is
 > that when non-ASCII characters were entered through a newsreader, they
 > were incorrectly converted in the Web view.  They look fine now.

The PHP version was very old. The newer ones fixed lotts of PHP bugs.

 > The Web view also added an extra space if text was indented in the
 > newsreader.  This is still happening.
Extra space? Where?

Thorsten

Post a reply to this message

On 3/23/21 4:27 AM, Thorsten wrote:
> On 22.03.2021 18:05, Cousin Ricky wrote:
>  > One thing that was wonky for a long time but seems to be fixed now is
>  > that when non-ASCII characters were entered through a newsreader, they
>  > were incorrectly converted in the Web view.  They look fine now.
> 
> The PHP version was very old. The newer ones fixed lotts of PHP bugs.
> 
>  > The Web view also added an extra space if text was indented in the
>  > newsreader.  This is still happening.
> Extra space? Where?
> 
> Thorsten

ah ok... you posting here reminded me of one other item i needed to look 
at is the multilink extension you implemented. i see 6 files in the 
tarball that chris made available. i don't see /any/ mysql calls that 
need to be converted to mysqli... appears you're doing it differently. 
want me to email you an archive of those files so you can 
review/bless/decide /not/ to implement your code.

Post a reply to this message

On 3/22/21 11:54 AM, Chris Cason wrote:
> Current status: the webview of the newsgroups is back up again, though 
> in read-only mode for now (and not taking a feed from the NNTP server yet).
> 
> There were a LOT of changes needed to the code to make it work with PHP7 
> so some breakage or strange behavior can be expected. If anything looks 
> particularly wonky please let me know.
> 
> -- Chris

wiki status update: i've found only 21 occurrence (in two files) of 
mysql->mysqli that needed to be converted. i've also been able to pull a 
file i created from the database, but have /not/ done an import. i'd 
like to resist doing that for now unless it's absolutely necessary. i'm 
feeling 99% ready to get at a live wiki so i can finish testing 
wikikdocgen

Post a reply to this message

On 2021-03-21 4:27 AM (-4), Thorsten wrote:
> On 22.03.2021 18:05, Cousin Ricky wrote:
> 
>  > The Web view also added an extra space if text was indented in the
>  > newsreader.  This is still happening.
> Extra space? Where?

https://news.povray.org/60467377%241%40news.povray.org

My first indent in the code excerpt is two spaces, but the Web view 
shows three spaces.  A 4 space indent shows as 5, etc.

Post a reply to this message

On 23.03.2021 13:28, Ash Holsenback wrote:
> ah ok... you posting here reminded me of one other item i needed to look 
> at is the multilink extension you implemented. i see 6 files in the 
> tarball that chris made available. i don't see /any/ mysql calls that 
> need to be converted to mysqli... appears you're doing it differently. 
> want me to email you an archive of those files so you can 
> review/bless/decide /not/ to implement your code.

Does it even still work? It was very fragile depending on MediaWiki 
version...

Thorsten

Post a reply to this message

On 23.03.2021 14:11, Cousin Ricky wrote:
 > On 2021-03-21 4:27 AM (-4), Thorsten wrote:
 >> On 22.03.2021 18:05, Cousin Ricky wrote:
 >>
 >>  > The Web view also added an extra space if text was indented in the
 >>  > newsreader.  This is still happening.
 >> Extra space? Where?
 >
 > https://news.povray.org/60467377%241%40news.povray.org
 >
 > My first indent in the code excerpt is two spaces, but the Web view
 > shows three spaces.  A 4 space indent shows as 5, etc.

Indeed, I can confirm this. I will see what is going on.

Thorsten

Post a reply to this message

On 23.03.2021 14:33, Thorsten wrote:
> On 23.03.2021 14:11, Cousin Ricky wrote:
>  > On 2021-03-21 4:27 AM (-4), Thorsten wrote:
>  >> On 22.03.2021 18:05, Cousin Ricky wrote:
>  >>
>  >>  > The Web view also added an extra space if text was indented in the
>  >>  > newsreader.  This is still happening.
>  >> Extra space? Where?
>  >
>  > https://news.povray.org/60467377%241%40news.povray.org
>  >
>  > My first indent in the code excerpt is two spaces, but the Web view
>  > shows three spaces.  A 4 space indent shows as 5, etc.
> 
> Indeed, I can confirm this. I will see what is going on.

Ok, so it turns out there are really three spaces in the message. The 
reason is, well, Thunderbird. You can see it in Thunderbird using 
More->View Source. It implements an email extension on a new server. The 
extension is called "format=flowed" and it is known to cause problems 
when transmitting source code over email (and thus news messages as 
well: 
https://www.kernel.org/doc/html/latest/process/email-clients.html#email-clients

Just google "format=flowed thunderbird" (without quotes). You will find 
plenty of complaints about it ...

Thunderbird (which I am also using) apparently has bugs with handling 
soft and hand line breaks and the correct conversion process to a final 
message format. It basically inserts the spaces even for hard line 
breaks, which makes exactly no sense at all when posting source code 
(which it cannot know, of course). It is only by chance that it then 
doesn't soft wrap your code when displaying the message...

Sorry,
Thorsten

Post a reply to this message

On 3/23/21 9:28 AM, Thorsten wrote:
> On 23.03.2021 13:28, Ash Holsenback wrote:
>> ah ok... you posting here reminded me of one other item i needed to 
>> look at is the multilink extension you implemented. i see 6 files in 
>> the tarball that chris made available. i don't see /any/ mysql calls 
>> that need to be converted to mysqli... appears you're doing it 
>> differently. want me to email you an archive of those files so you can 
>> review/bless/decide /not/ to implement your code.
> 
> Does it even still work? It was very fragile depending on MediaWiki 
> version...
> 
> Thorsten

yes it was working before the crash... however a /big/ performance hit 
when editing/saving the reserved keywords page (it's loaded with 
multi-links). having said that... i'm in favor of backing this extension 
out. not sure what the busted link fallout will be. i'm hoping isolated 
because i've /not/ been using the format. lol... remember i was /not/ 
pleased with the implementation, because at the release (last minute) i 
had to find a way to deal with those links. i think i /might/ be able to 
coax wikidocgen to help find them because i already have a case test in 
the code.

Post a reply to this message

On 2021-03-23 9:55 AM (-4), Thorsten wrote:
> On 23.03.2021 14:33, Thorsten wrote:
>> On 23.03.2021 14:11, Cousin Ricky wrote:
>>  > On 2021-03-21 4:27 AM (-4), Thorsten wrote:
>>  >> On 22.03.2021 18:05, Cousin Ricky wrote:
>>  >>
>>  >>  > The Web view also added an extra space if text was indented in the
>>  >>  > newsreader.  This is still happening.
>>  >> Extra space? Where?
>>  >
>>  > https://news.povray.org/60467377%241%40news.povray.org
>>  >
>>  > My first indent in the code excerpt is two spaces, but the Web view
>>  > shows three spaces.  A 4 space indent shows as 5, etc.
>>
>> Indeed, I can confirm this. I will see what is going on.
> 
> Ok, so it turns out there are really three spaces in the message. The
> reason is, well, Thunderbird. You can see it in Thunderbird using
> More->View Source. It implements an email extension on a new server. The
> extension is called "format=flowed" and it is known to cause problems
> when transmitting source code over email (and thus news messages as
> well:
> https://www.kernel.org/doc/html/latest/process/email-clients.html#email-clients
> 
> 
> Just google "format=flowed thunderbird" (without quotes). You will find
> plenty of complaints about it ...
> 
> Thunderbird (which I am also using) apparently has bugs with handling
> soft and hand line breaks and the correct conversion process to a final
> message format. It basically inserts the spaces even for hard line
> breaks, which makes exactly no sense at all when posting source code
> (which it cannot know, of course). It is only by chance that it then
> doesn't soft wrap your code when displaying the message...

OK, thanks.  I just turned it off on my end.  I'll see how that works
out in the future.

Post a reply to this message

Folks,

I had to re-build the NNTP server indexes and caches again while trying 
to diagnose a feed issue. This may cause the need to fetch headers again 
(at least for those who had this problem last time I needed to do this).

-- Chris

Post a reply to this message

Current status: we've run into an issue with the NEWNEWS NNTP command on 
the new architecture and have had to disable it. Since this was the way 
the web view got postings into the DB I've had to work around it. The 
new method works fine but isn't automated yet. I ran it once to import 
all the posts that were made since the server came back up; these are 
now available in the webview. I'll automate it in the next day or so, 
until then new posts won't turn up.

I also noticed an issue with posts containing high ASCII (e.g. hex A0, 
which is a non-breaking space). With everything running under UTF-8 
encoding now, PHP considers these as invalid code sequences when calling 
htmlentities() or htmlspecialchars(), leading to blank lines anywhere 
one is present (e.g. http://news.povray.org/60567d33@news.povray.org has 
these in the blank section of quoted text you see. If you don't see a 
blank section of quoted text, it means we've now fixed it :)

This will likely manifest it as strangeness in other parts of the 
webview since invalid code sequences in names, titles, etc have the 
potential to get turned into blank strings.

There is a workaround for it (ENT_SUBSTITUTE, which looks fugly, or 
ENT_IGNORE (which may have security implications). We'll work out what 
to do and let you know. In the meantime if you spot anything that shows 
this issue other than in message bodies please LMK.

-- Chris

Post a reply to this message

On 3/26/21 8:44 AM, Chris Cason wrote:
In the meantime if you spot anything that shows
> this issue other than in message bodies please LMK.
> 
> -- Chris

Is this related to the message digests?   I can see recent posts in the 
actual groups, but the message digests are still pre-crash.

Post a reply to this message

On 26.03.2021 18:44, Bald Eagle wrote:
> On 3/26/21 8:44 AM, Chris Cason wrote:
> In the meantime if you spot anything that shows
>> this issue other than in message bodies please LMK.
>>
>> -- Chris
> 
> Is this related to the message digests?   I can see recent posts in the 
> actual groups, but the message digests are still pre-crash.

No,can't be. the digests are dynamically generated. So what you see 
would be a bug.

The problem with non-ASCII is related to the codebase being pre-propper 
UTF8 support pretty much everywhere, and now having to work with code 
that does UTF8 properly. The good news is that this means that most 
likely all known problems will get fixed now that developed over the 
past two decades, as I will be reviewing the whole code for UTF8 issues 
in the next few days (unlike Chris, I have to deal with UTF8 more 
frequently in Germany...)

Thorsten

Post a reply to this message

On 27/03/2021 04:44, Bald Eagle wrote:
> Is this related to the message digests? I can see recent posts in the
> actual groups, but the message digests are still pre-crash.

Digests seem to be working for me (or at least are not the pre-crash
version). Try a shift-reload on the page in question to force flushing
of your browser cache, see if it makes a difference. If not please LMK
the particular page you see this on.

-- Chris

Post a reply to this message

Current status: posts are now auto-importing into the web view DB. The 
schedule is as before (every 5 minutes).

Additionally posting from the web view has now been re-enabled; those 
who use that interface may like to test some posts and attachments. 
Please LMK of any issues.

-- Chris

Post a reply to this message

hi,

Chris Cason <del### [at] deletethistoopovrayorg> wrote:
> Current status: posts are now auto-importing into the web view DB. The
> schedule is as before (every 5 minutes).
>
> Additionally posting from the web view has now been re-enabled; those
> who use that interface may like to test some posts and attachments.
> Please LMK of any issues.
>

first off, thank you + all involved for your work, for the continuity.

I've been using the RSS feed to read new posts (on Android), but even after
refresh (and re-subscribe a few minutes ago) I see the date/time of posts but no
content.


regards, jr.

Post a reply to this message

On 28/03/2021 00:11, jr wrote:
> refresh (and re-subscribe a few minutes ago) I see the date/time of posts but no
> content.

I'll have a look it.

-- Chris

Post a reply to this message

On 28/03/2021 00:11, jr wrote:
> I've been using the RSS feed to read new posts (on Android), but even after
> refresh (and re-subscribe a few minutes ago) I see the date/time of posts but no
> content.

Should be fixed now.

-- Chris

Post a reply to this message

hi,

Chris Cason <del### [at] deletethistoopovrayorg> wrote:
> On 28/03/2021 00:11, jr wrote:
> > ... RSS ...
>
> Should be fixed now.

looks good, thank you.


regards, jr.

Post a reply to this message

On 22/03/2021 01:46, Bald Eagle wrote:
> Send it over.   I'll have a look and see if I have any ideas.
> Seems like 2021 is the POV-Ray Year of the Update.  ;)

When I got to the point of looking at the HTTP config for tag.povray.org 
I discovered something that I'd forgotten I did: the URL of the tag Q&T 
pages were long ago redirected to a re-hosted form of them on the wiki.

The link on the povray.org menu still pointed at tag.povray.org but 
anyone clicking on it would have ended up in the wiki.

Since I had forgotten I did this when bringing the main site back up 
(and hadn't done TAG yet) I thought it was still live on the TAG page :-/

So thanks for the offer, it won't be needed for the HTML version of the 
Q&T anyhow. I'll keep the current version until we get the wiki back up.

-- Chris

Post a reply to this message

On 23/03/2021 23:41, Ash Holsenback wrote:
> wiki status update: i've found only 21 occurrence (in two files) of 
> mysql->mysqli that needed to be converted. i've also been able to pull a 
> file i created from the database, but have /not/ done an import. i'd 
> like to resist doing that for now unless it's absolutely necessary. i'm 
> feeling 99% ready to get at a live wiki so i can finish testing wikikdocgen

Thanks ... with most of the infrastructure now back up and running I can 
now start looking at how I will re-host the wiki and lib sites. I'll let 
you know when I make progress (probably via email).

-- Chris

Post a reply to this message

On 22/03/2021 01:28, Bald Eagle wrote:
> newsgroups work.   I know that a lot of us would like to be able to 
> include attachments in ANY of the newsgroups.

Try it now :>)

> possible.  I know that Mike Williams' site has been down for a while. I 
> backed that up and created a PDF and a zip file of all of his sources. I 
> have all of Friedrich Lohmueller's excellent work as well.
> Perhaps we could have some dedicated space on the new server for that 
> material,

I'm happy to re-host that and other related resources provided that 
copyright / license issues don't get in the way.

> and maybe mull over a mechanism by which the core POV-Ray 
> material (source, binaries, documentation) could have backups hosted on 
> one or more other sites.

archive.org kind of does this for the www site in an informal way. That 
said though we always designed the site such that everything important 
sits in our FTP directories. For example, ftp://ftp.povray.org/pub/ and 
below contains binaries going back to version 1.0. The same physical 
directory tree is mapped to the WWW site as http://www.povray.org/ftp/, 
and this is where downloads get pulled from when initiated from the www.

The reason for doing it this way is that in the past (much less so now, 
though) it was common for FTP sites to be mirrored in various places 
around the world (less as a means of backup than as a speedup for local 
users as international bandwidth wasn't what it is now, and also as a 
means of offloading traffic from a main site).

We used to list the formal mirrors (those we had arrangements with) on 
our download page but don't anymore as they tend to come and go more 
than they used to. You also could find mirrors via the Archie search 
engine but I don't think that's really a thing anymore.

That said there are still mirrors out there, e.g. 
https://www.mirrorservice.org/sites/ftp.povray.org. I'm sure there's 
more but since FTP sites themselves aren't as far as I am aware directly 
indexed by web search engines they will only turn up in searches if the 
FTP site also has a HTTP front-end.

> specialized technical work.  Everything associated with the new server 
> is not free - Perhaps there is something some of us could do by check, 
> paypal, patreon, ....?

Appreciate the thought, but not needed at this point.

-- Chris

Post a reply to this message

On 22/03/2021 01:28, Bald Eagle wrote:
> material, and maybe mull over a mechanism by which the core POV-Ray 
> material (source, binaries, documentation) could have backups hosted on 
> one or more other sites.

FWIW I am considering using cloudflare to optimize access to the site. 
Apart from better response overall they also have the ability to keep 
parts of a site online when the origin server (i.e. us) drops out via 
their "Always Online" feature 
(https://support.cloudflare.com/hc/en-us/articles/200168436).

Their free account would probably work OK but there's more functionality 
available to us on pro or upwards. They do offer free pro accounts to 
eligible open-source projects so I've applied for this (it's not clear 
if we're eligible, but time will tell).

If not then I can still do stuff with the free tier, I'll have to get it 
set up and see.

-- Chris

Post a reply to this message

Chris Cason <del### [at] deletethistoopovrayorg> wrote:
> On 22/03/2021 01:28, Bald Eagle wrote:
>
> > include attachments in ANY of the newsgroups.
>
> Try it now :>)

I'll give it whirl as soon as I'm able.  :)
DMV after work today   :(

> I'm happy to re-host that and other related resources provided that
> copyright / license issues don't get in the way.

I guess I'll see what I can find.
I've run into a couple of neat links via archive.org, and to be honest, most of
the sites are defunct, OLD (thus the internet archaeology required), outdated
(POV-Ray 3.5 or earlier), and the authors may very well no longer be with us in
this realm.

To Quote Joshua English when he replied to my email in 2016, "Wow. I didn't
think anyone actually could find the Cyclopedia after all these years."

Well now I can't - because his site is down too --- so I'll be HHTracking that
one too.

From an archaeological and historical point of view, some of the material is of
important value, and the authors would be pleased to have it shared and
disappointed and bewildered if it wasn't. Making it available for people to
learn from and USE was the whole point of putting in the creative work to begin
with.
From a practical standpoint - they posted it on the web (maybe in 1998), it got
archived, I can access it.   Who actually hosts it seems a bit irrelevant.

I think is we slap a (c) Someguyon Theinternet notice on it, with a "contact us
if you think this violates xyz, and want it removed" boilerplate, all will be
good.
I'd be positively flabbergasted if someone('s next of kin) took issue with, let
alone _FOUND_ a resurrected copy of a 23 year old POV-Ray include file or
tutorial.

I will post a list of what I have - hopefully by the end of the week, and you
can decide how such things will be integrated into the site.

Post a reply to this message

Chris Cason <del### [at] deletethistoopovrayorg> wrote:

>
> Additionally posting from the web view has now been re-enabled; those
> who use that interface may like to test some posts and attachments.
> Please LMK of any issues.
>
> -- Chris

A massive THANK YOU to you, Thorsten, Ash and others here for tracking down and
solving so many issues, and for getting all of this stuff up and running again.
All of your hard work is *definitely* appreciated.

I'm using the web portal, and it all looks good so far. For what it's worth, as
of 45 minutes ago, I still couldn't post a reply here (I was seeing a blank
screen when I hit "reply to this post".) But now it works! GREAT!

Congratulations!!!

Post a reply to this message

"Kenneth" <kdw### [at] gmailcom> wrote:
> Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> > Additionally posting from the web view has now been re-enabled; those
> > who use that interface may like to test some posts and attachments.
> > Please LMK of any issues.

I have only tested the newsgroups so far, not the rest of the site / links.
A recent newsgroup question prompts me to ask: will multiple attachments be
possible through the web interface, (and will any more wiki accounts be
available for use)?

> A massive THANK YOU to you, Thorsten, Ash and others here for tracking down and
> solving so many issues, and for getting all of this stuff up and running again.
> All of your hard work is *definitely* appreciated.

A hearty second to that.  :)   I must admit that I was on the verge of panic
when the site went down ... and stayed down...  :O

Here's to another 25+ years of raytracing - the old skool way.  :D

Post a reply to this message

On 1/04/2021 09:32, Bald Eagle wrote:
> A recent newsgroup question prompts me to ask: will multiple attachments be
> possible through the web interface,

I guess it could be. I've not looked at that side of it for a while.

> (and will any more wiki accounts be available for use)?

I can't imagine we'd be able to open it up to the world again since 
we're just constantly fighting spammers. But that doesn't mean we can't 
hide the registration page in a non-standard spot* and only pass that on 
to folks we want to allow registration from. I'll leave it up to Ash.

* or require a password just to access the page, via .htaccess.

-- Chris

Post a reply to this message

This part of the site is still down:

http://lib.povray.org/

I think someone moved all that stuff to GitHub anyway, but now I can't 
find it there. Was the repo deleted?


Mike

Post a reply to this message

On 2/04/2021 12:05, Mike Horvath wrote:
> This part of the site is still down:
> 
> http://lib.povray.org/

Yep, that and the wiki ran on a sub-server (kind of a virtual machine). 
I haven't had a chance to set that up again.

> I think someone moved all that stuff to GitHub anyway, but now I can't 
> find it there. Was the repo deleted?

Not sure, hadn't heard of that myself.

-- Chris

Post a reply to this message

On 2021-04-01 9:05 PM (-4), Mike Horvath wrote:
> This part of the site is still down:
> 
> http://lib.povray.org/
> 
> I think someone moved all that stuff to GitHub anyway, but now I can't
> find it there. Was the repo deleted?

They are at https://github.com/LeForgeron/PovContributions

Jerome updates these manually, copying them from lib.povray.org.

Post a reply to this message

No idea if this is related to the utf-8 issues.
I was trying to answer someone else's question and came across this old post:

http://news.povray.org/povray.advanced-users/message/%3Cweb.57a001e45c93b6e65e7df57c0%40news.povray.org%3E/#%3Cweb.57a0
01e45c93b6e65e7df57c0%40news.povray.org%3E

I don't recall it looking like that when it was originally posted ....

Post a reply to this message

On 3/04/2021 04:54, Bald Eagle wrote:
> No idea if this is related to the utf-8 issues.
> I was trying to answer someone else's question and came across this old post:
> 
>
http://news.povray.org/povray.advanced-users/message/%3Cweb.57a001e45c93b6e65e7df57c0%40news.povray.org%3E/#%3Cweb.57a0
> 01e45c93b6e65e7df57c0%40news.povray.org%3E
> 
> I don't recall it looking like that when it was originally posted ....

Interesting find. I've looked at the raw message on the NNTP server and 
it does in fact have the entity encoding present; e.g.

   x = & #961;cos(& #952;)sin(& #934;)
   y = & #961;sin(& #952;)sin(& #934;)
   z = & #961;cos(& #934;)

(Note I have spaced these out so they won't be classed as HTML entities 
and re-encoded by the server. The original had no spaces). This is how 
it would have appeared to NNTP users. The HTML entities are due to it 
being posted via the web interface and were sucked into the NNTP side as-is.

When re-displaying them, PHP's htmlentities() function is (correctly) 
escaping them as it assumes the input is not HTML formatted.

There is a way of telling htmlentities() to not re-encode existing 
encoded sequences, and I thought that may have been a default value 
change between the old and new PHP, but the docs don't say so, and 
google's cache from 4 March (so before the crash) shows it with the 
escaping also:

http://webcache.googleusercontent.com/search?q=cache:YzzVZteLENYJ:news.povray.org/web.57a001e45c93b6e65e7df57c0%2540news.povray.org

so I suspect it's probably been like that for a while. That said, 
though, it is an issue - either we ought to decode entity-encoded HTML 
before putting it into the NNTP server, or we need to not double-encode 
them when pulling out for display.

Given the data is already stored I can't change the first option, so for 
now I've turned on the second one (existing HTML encoding is passed over).

I'll need to have a think about what implications this might have going 
forward as I'm not sure it's the right fix (e.g. perhaps to only apply 
it to posts that were originally made via the web interface).,

-- Chris

Post a reply to this message

On 2/04/2021 12:05, Mike Horvath wrote:
> This part of the site is still down:
> 
> http://lib.povray.org/

I've had a stab at updating the lib.povray.org PHP code but have found 
it's going to be more work than I can allocate time to right now. I'll 
look at it later.

-- Chris

Post a reply to this message

wiki.povray.org is now back online, as is bugs.povray.org (not used for 
new bugs, I brought it back mainly for historical reasons).

The wiki had quite a few software changes, including a significant 
re-factoring of a custom extension we had, so if you spot anything out 
of place please LMK (in particular, the extensions for syntax 
highlighting and LaTeX markup have changed).

-- Chris

Post a reply to this message

in news:6072f98d$1@news.povray.org Chris Cason wrote:

> wiki.povray.org is now back online

Thanks!

Ingo

Post a reply to this message

Op 11-4-2021 om 16:00 schreef ingo:
> in news:6072f98d$1@news.povray.org Chris Cason wrote:
> 
>> wiki.povray.org is now back online
> 
> Thanks!
> 
> Ingo
> 

and thanks too!

-- 
Thomas

Post a reply to this message

in news:604ec2df$1@news.povray.org Chris Cason wrote:

> we've only lost about one 
> day of data

Over the years I've lost my newsgroup archive several times and currently 
don't maintain it. Could it be made available as a (always up to date) 
download in some way? 
I'd like to try and search it with SQLite's text indexing features.

Ingo

Post a reply to this message

Since someone is working on the server, when searching on Google, the 
wiki doesn't always show up in search results. For instance:

https://www.google.com/search?q=povray+average+pigment

Is there any way the server admin can demote the regular docs pages and 
promote wiki pages in results?


Thanks.

Mike

Post a reply to this message

Am 15.03.2021 um 03:10 schrieb Chris Cason:

> The NNTP server is the first to be brought back online and at the time I 
> post this is the only service running and publicly accessible. Posts 
> made here will eventually migrate to the web view once that's sorted and 
> back up; in the meantime only NNTP users will see this message.

Just trying to slowly get back up to speed, reading up on what's been 
happening while I wasn't looking, and found that Thunderbird refused to 
load a suspiciously large number of messages from back then, saying they 
didn't exist anymore. Tried to repair the index, and now all messages 
from 2019 seem to have been gone entirely (as well as a few months 
before and after).

Could that be related to the March 9 server failure, or was there an 
earlier separate problem that ate all the posts?

Post a reply to this message

On 26/05/2021 06:05, clipka wrote:
> Just trying to slowly get back up to speed, reading up on what's been 
> happening while I wasn't looking, and found that Thunderbird refused to 
> load a suspiciously large number of messages from back then, saying they 
> didn't exist anymore. Tried to repair the index, and now all messages 
> from 2019 seem to have been gone entirely (as well as a few months 
> before and after).
> 
> Could that be related to the March 9 server failure, or was there an 
> earlier separate problem that ate all the posts?

Any group in particular that is missing posts, or is it all?

For povray.general at least, posts seem to be there ...

   telnet news.povray.org 119
   group povray.general
   211 54252 3332 83275 povray.general selected

Estimated 54,252 articles in total, first article 3332 and last is 
83275. I'm able to retrieve both the first (from 1998) and the last 
(from today).

However that's not to say I know what exact requests Thunderbird is 
sending. If it happens again I suggest running tcpdump or wireshark on 
your end when you do the fetch so you can see exactly what Thunderbird 
is asking and what the server tells it in response. If you can send that 
to me I can take a look at it.

-- Chris

Post a reply to this message

Am 26.05.2021 um 02:06 schrieb Chris Cason:

> Estimated 54,252 articles in total, first article 3332 and last is 
> 83275. I'm able to retrieve both the first (from 1998) and the last 
> (from today).

That about matches the number of posts I'm seeing now, after rebuilding 
the index (54243; not sure where the difference of 10 is from). Before 
though, there were something like 70k articles for which Thunderbird had 
retrieved header information.

Posts as far back as 1998 are fine, as are recent posts. What I'm 
SPECIFICALLY NOT seeing anymore are any posts dating between 2018-10-07 
and 2020-04-18. And I know there were quite a lot, even a couple I had 
read just before rebuilding the index. Even a couple I had posted myself.

The same happens in each and every other group: Any posts from 2019 that 
are still in the index over here, but that I hadn't read yet, all just 
prompt a message saying they no longer exist. And if I rebuild the index 
of any group, all posts from 2019 - including ones I had read just 
minutes before - are just gone, without any trace whatsoever.

The web interface still seems to have them, for some reason. But the 
news server apparently doesn't.

> However that's not to say I know what exact requests Thunderbird is 
> sending. If it happens again I suggest running tcpdump or wireshark on 
> your end when you do the fetch so you can see exactly what Thunderbird 
> is asking and what the server tells it in response. If you can send that 
> to me I can take a look at it.

I have no idea what tcpdump or wireshark even do (I may be exaggerating 
here, but not by much), let alone how to run them on my Windows machine 
to put them to any good use.

Post a reply to this message

On 2021-05-26 8:40 AM (-4), clipka wrote:
> 
> Posts as far back as 1998 are fine, as are recent posts. What I'm
> SPECIFICALLY NOT seeing anymore are any posts dating between 2018-10-07
> and 2020-04-18. And I know there were quite a lot, even a couple I had
> read just before rebuilding the index. Even a couple I had posted myself.
> 
> The same happens in each and every other group: Any posts from 2019 that
> are still in the index over here, but that I hadn't read yet, all just
> prompt a message saying they no longer exist. And if I rebuild the index
> of any group, all posts from 2019 - including ones I had read just
> minutes before - are just gone, without any trace whatsoever.
> 
> The web interface still seems to have them, for some reason. But the
> news server apparently doesn't.

I have the same problem.

Post a reply to this message

Op 26-5-2021 om 16:12 schreef Cousin Ricky:
> I have the same problem.
> 
As do I also.

-- 
Thomas

Post a reply to this message

On 26/05/2021 22:40, clipka wrote:
> The web interface still seems to have them, for some reason. But the 
> news server apparently doesn't.

Sounds like the server has a problem with re-indexing. I know it did 
have an issue with XOVER for more recent dates and hence I had to turn 
that off.

If it turns out 2018-10-07 refers to some 'special' number in 32-bit 
seconds since 1970 (or close to one) then it's likely an internal issue. 
If it doesn't then I'll have to dig further.

If it turns out I can't fix it I'll have to change to another NNTP 
server (and that assumes I can find a way to get the missing messages 
imported).

The message spool files are basically large text blobs so are readble 
and I know I've had to do that once before maybe 20 years ago. But 
that's a last resort as the spam filtering the server uses is a script 
custom-written for the current server. If e.g. INN supports spam filters 
I expect there will be a way to get it ported.

-- Chris

Post a reply to this message

Am 31.05.2021 um 05:14 schrieb Chris Cason:

> If it turns out 2018-10-07 refers to some 'special' number in 32-bit 
> seconds since 1970 (or close to one) then it's likely an internal issue. 
> If it doesn't then I'll have to dig further.

2018-10-07 00:00 UTC would be 1539734400 seconds since beginning of the 
Unix epoch (i.e. 1970).

The magic rollover for 32-bit signed values (or 31-bit unsigned) will be 
in early 2038.

The magic rollover for 1 bit less would have been in early 2004.

So no, that doesn't quite fit the symptoms.

Post a reply to this message

On 26/05/2021 22:40, clipka wrote:
> Posts as far back as 1998 are fine, as are recent posts. What I'm 
> SPECIFICALLY NOT seeing anymore are any posts dating between 2018-10-07 
> and 2020-04-18. And I know there were quite a lot, even a couple I had 
> read just before rebuilding the index. Even a couple I had posted myself.

I can confirm I see this behavior locally now also. It's not *every* 
message, but certainly a lot of them, predominantly high-traffic groups.

In terms of what's causing it, I do note that the last message in one of 
the internal spool files (db_312.itm) is dated 7 Oct 2018 17:36:06. The 
next message in that thread, according to the web view, is 
<5bbacb83$1@news.povray.org>, dated 7 Oct 2018 23:14:11.

The next db_*.itm file, sequentially, is db_314.itm (there's no 313), 
however it contains posts from 2006. A grep through all the *.itm files 
shows that <5bbacb83$1@news.povray.org> is, in fact, missing entirely.

(The db_*.itm files aren't always sequential; some are, but sometimes it 
starts writing to a different sequence. I'm not sure of the trigger for 
this, but suspect it is intentional).

I did the same grep on a backup I have locally of the server prior to 
the crash and DID find the article: it's in db_1.itm, and in fact is 
also the very first article.

So: at some point on 7 October 2018 the news server switched from 
writing to db_312.itm to db_1.itm. There's nothing in the command log 
file indicating I issued any particular command (e.g. re-index) on that 
day, nor do the server logs show a reboot or anything interesting, and 
finally the news server read log for the day shows articles being read 
before and after the cutoff time right through the evening without any 
apparent major interruption or break.

Returning to this year: grepping the nntp server log revealed that on 25 
March during a re-index the server decided that db_1.idx (and a bunch of 
others) was 'empty lost' (whatever that means) and nuked them. 
Subsequently (31 March) it created a new db_1.itm and filled that with 
new articles as they came in.

So ... the server nuked a bunch of article files for a reason that is 
not clear and has subsequently started re-using the db sequence numbers.

The good news is I have all the removed .itm files, other than the 
below, in a backup. The exception is that on Mar 27, Mar 31 and May 6 
the server also nuked, respectively, db_0.itm, db_1.itm and db_2.itm 
again, and these have subsequently been re-used. As these were 
post-crash I only have the current versions of them in my backup.

TL;DR a bunch of items were lost. I can try to restore them by copying 
over the nuked .itm files from a backup, re-naming them where necessary, 
in the hope that when I re-start the news server it will ingest them 
rather than deleting them. I'll experiment with this on a test server 
rather than risk breaking this one more.

-- Chris

Post a reply to this message

Following on from my previous post about the cause of the missing articles:

In the long term given I really don't have any insights into how our 
current news server does stuff internally (it's not open-source) and the 
fact it's old and unsupported I think I have no choice but to migrate to 
a different server (probably INN).

I should be able to code a means of converting articles between the 
respective spool formats to avoid needing to have the new server take a 
feed from the current one (this would add an unnecessary new component 
to the path of each article, amongst other possible tweaks).

For any articles still missing after I process the current spool and the 
previously nuked but saved db_*.itm files, I should be able to fetch 
them from the webview database. Once done I just need to port the spam 
filter and I should then be able to bring the new NNTP server up.

I will, if I can find a way to do it, attempt to keep the article 
numbers within the groups identical. If that's not possible, though, a 
full re-fetch of the groups will be necessary for NNTP users.

The priority I place on doing this will depend on whether or not I can 
get the missing articles back into the current server without it 
throwing a hissy fit and deleting them again.

-- Chris

Post a reply to this message

Thank you very much for trying to fix this Chris !

--
Tor Olav

Post a reply to this message

On 15/04/2021 16:30, ingo wrote:
> Over the years I've lost my newsgroup archive several times and currently
> don't maintain it. Could it be made available as a (always up to date)
> download in some way?

If I end up migrating to another news server I will need to collate all 
the messages I can find (including getting any missing ones from the 
webview DB).

If I do that I could provide the collated messages in some form of 
download. I may be able to keep that updated by appending to it new 
messages as the come in, provided there's enough interest in it.

One complication of a simple append scheme, though, is that 
(intentionally) deleted messages won't get removed, and I think that any 
archive of this sort should have deleted messages taken out of it.

It may be better to just dump the stored messages from the DB as they 
will always reflect only undeleted messages. I'll decide when I look at 
the migration.

BTW it's occurred to me that one useful outcome of having a downloadable 
archive of this sort is that we could auto-submit it to archive.org on a 
regular basis.

-- Chris

Post a reply to this message