On 14/02/2023 20:59, William F Pokorny wrote:
> To chime in with a related topic; Many of us were using Thunderbird's search
facilities of the newsgroups over the formally less reliable web based search / google
search.
> 
> As, I think, you are aware, after the recovery from the crash the news server is
maintaining much less of the overall posting history. It's often impossible to get to
any older posts these days via Thunderbird.
> 
> I remember you saying the POV-Ray news server had been somewhat mangled in the past
to keep posts in most, but not all forums, for a very long time. Something much longer
the usual 12 months or so. My 'guess' is that when the news server was restored most,
if not all, news server forums are now dropping posts after a much shorter time (12
months?).
> 
> Whether this worth fixing or not I'm unsure. Expect these days the newsgroup
interface not the primary one for most users. Having a reliable way to search the
forums should be primary the goal.

I don't think the articles are being deleted. I do suspect that it's difficult or
impossible to get the NNTP server to give a reliable view of what messages are
available.

I'm not sure if it's because I kept my old Thunderbird NNTP cache across the crash or
whether I was lucky and it did download all the headers from the new server before it
started playing up. As it is, though, thunderbird shows me messages in
povray.binaries.images going back to 2005. In povray.bugreports it shows messages
going back to 1998, though if I try to load them it says there's no such article. The
retained articles (where I was able to get it to load from the server) in this group
seem to start around September 2009.

In PBI I have messages going back to 2005 and they all seem to load when I do spot
checks.

To test that an article really is on the server and not just locally cached you can
easily check using telnet as long as you know the article ID. Example:
  
> # telnet news.povray.org 119
> Trying 203.29.75.35...
> Connected to news.
> Escape character is '^]'.
> 200 news.povray.org DNEWS Version  5.7b1,, S0, posting OK

Ask for for an old post where I know the article ID:

> article <429e54b3$1@news.povray.org>
> 220 58213 <429e54b3$1@news.povray.org> article retrieved - head and body follows
> From: "trulayne" <tru### [at] newrrcom>
> Newsgroups: povray.general
> Subject: Equally spaced objects along a spline curve
> Date: Wed, 1 Jun 2005 19:37:06 -0500
[SNIP]
> TIA
> Dennis
> .

Select PBI as the current group:

> group povray.binaries.images
> 211 74934 66649 146750 povray.binaries.images selected

This tells us the oldest message is article number 66649 and the newest is 146750
(this is the message I'm replying to).

Check what 66649 was:

> head 66649
> 221 66649 <3d6bd206$1@news.povray.org> article retrieved - head follows
> From: "Zeger Knaepen" <zeg### [at] studentkuleuvenacbe>
> Newsgroups: povray.binaries.images
> References: <3d6b4d0b@news.povray.org> <3d6baacc@news.povray.org>
<3d6bb72e@news.povray.org> <3d6bbf6a$1@news.povray.org>
> Subject: Re: Odd blur effect: squared spheres
> Date: Tue, 27 Aug 2002 21:24:35 +0200
[SNIP]
> Xref: news.povray.org povray.binaries.images:66649
> .

Now check the last message, 146750

> head 146750
> 221 146750 <63eb5b90$1@news.povray.org> article retrieved - head follows
> Date: Tue, 14 Feb 2023 04:59:44 -0500
> MIME-Version: 1.0
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
>  Thunderbird/102.7.1
> Subject: Re: Denoising POV-Ray images in Blender
> Newsgroups: povray.binaries.images
> References: <63dd0bd3$1@news.povray.org>
>  <web.63dd555f13ac33281f9dae3025979125@news.povray.org>
>  <63eb3670@news.povray.org>
> Content-Language: en-US
> From: William F Pokorny <ano### [at] anonymousorg>
> In-Reply-To: <63eb3670@news.povray.org>
[SNIP]
> Xref: news.povray.org povray.binaries.images:146750
> .
> quit
> 205 closing connection - goodbye!
> Connection closed by foreign host.

AFAICT all the messages are still in the news server but perhaps it's not presenting
that to thunderbird in a way that it understands, at least in some cases?

Even if they were being deleted, they aren't lost forever as I have a backup with all
the posts prior to the crash and also each message is stored in the SQL db used for
the web view. Being able to *find* them is another issue, as google aren't being all
to helpful.

-- Chris

Post a reply to this message

Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> If I did it should solve the problem for local searches. However at the moment
fixing the google search results has h
igher priority since that's what a typical POV-Ray user would use to ask about
something. If it doesn't index all pages
 we would have cases where a user searched for something and while the answer actually
exists and is in our groups, goo
gle hasn't included it, so they don't get a relevant result.

.....

> As far as I can tell the issue is Google isn't indexing *all* of our pages,
regardless of what I do (for example I tr
ied using sitemaps in various forms and it still didn't fetch all the URL's in the
sitemap). I ran out of time trying t
o sort this and just left it alone, hoping google would catch up eventually.

I would take a poll to confirm your assertion about typical methods and thus
priority.

I don't trust Google or any of the other Big Tech companies _at all_ anymore.
For years, I have experienced finding a result and then looking for the exact
same thing the very next day to share the link, and it doesn't come up in the
results AT ALL.  The search results are constantly changing, getting filtered,
and being manipulated.

If you make a reliable search feature, people would use that and then that would
de facto become the typical method for finding relevant posts - in a timely
manner.   Either way, it's a self-fulfilling argument.

Post a reply to this message

On 2/14/23 06:25, Chris Cason wrote:
> AFAICT all the messages are still in the news server but perhaps it's 
> not presenting that to thunderbird in a way that it understands, at 
> least in some cases?

The PBI group most reliably reports older posts my thunderbird goes back 
to 2002, but not all the post show up as actually on the server. Plus, 
as a reference I'm less interested in this group than others.


Moving to another group I'd like to be cleanly working for me:

povray.beta-test.binaries (where I was posting many of my ideas)
-------------------------

Post:
Function / Pattern issues. New function f_eblob().  11/22/19

Thunderbird gives me:

Article not found

The newsgroup server reports that it can't find the article.

     Newsgroup server responded: no such article number in this group
     Perhaps the article has expired?
     Try searching for article: <5dd8006c$1@news.povray.org> (574)

When I try your telnet suggestion:

pokorny:tmp$ telnet news.povray.org 119
Trying 203.29.75.35...
Connected to news.povray.org.
Escape character is '^]'.
200 news.povray.org DNEWS Version  5.7b1,, S0, posting OK
article <5dd8006c$1@news.povray.org>
430 no such article found

---

I can get / see via thunderbird:

Function / Pattern issues. Adding f_granite23,...  04/19/20

The id is: 5e9cce19@news.povray.org

pokorny:tmp$ telnet news.povray.org 119
Trying 203.29.75.35...
Connected to news.povray.org.
Escape character is '^]'.
200 news.povray.org DNEWS Version  5.7b1,, S0, posting OK
article <5e9cce19@news.povray.org>
... I get what looks like some large mime dump that I suspect is
due some attached images... Probably the header is there, but not sure 
how to see the header in this case.

If I try exactly what you posted for the first article command, I see 
more or less exactly what you posted.

---

Trying:
group povray.beta-test.binaries
211 173 199 662 povray.beta-test.binaries selected
head 199
21 199 <l5u60u4ak2fij7ocu5je8minjdemjc97mo@4ax.com> article retrieved - 
head follows
From: W�odzimierz ABX Skiba <abx### [at] babilonorg>
Newsgroups: povray.beta-test.binaries
Subject: Re: bug with bozo pattern
Date: Tue, 27 Nov 2001 12:31:39 +0100

head 662
Is OK too. It's the last time I posted to this group moving to the v4.0 
thread as a better fit for ideas given v3.8's intent to release as 
backed up by Christoph.

---

What to make of this? Some of my stuff is on the server and other stuff 
is not?

Bill P.

Post a reply to this message

Bald Eagle wrote:
> I would take a poll to confirm your assertion about typical methods and
> thus priority.

And William P wrote:
> I remember you saying the POV-Ray news server had been somewhat mangled
> in the past to keep posts in most, but not all forums, for a very long
> time. Something much longer the usual 12 months or so. My 'guess' is
> that when the news server was restored most, if not all, news server
> forums are now dropping posts after a much shorter time (12 months?).

For what it's worth: To search for old posts in Windows via Firefox, I always
use the web portal's 'search' button at the top of a typical newsgroup post,
which brings up the "enhanced by Google" entry box. I've never used Thunderbird
et al.

I just did a search for 'Kenneth' (by itself) and see that Google presents 10
pages of old entries MAXIMUM, with 10 hits per page. That's only 100 hits. (I've
been posting since...2005?)

Searching by 'date', the most recent entry is from January 2023. Wrong.

The oldest entry is from 2021. Very wrong.

Of course, I've done other searches, using my own name PLUS some relevant topic,
and get much-earlier-dated hits. Although, I really don't know if Google is
'missing' some of my old posts, as I haven't tried such a thorough test yet.

So one problem I see is that the web portal's search engine produces only 10
pages of hits at a time, no more. Is that a limit imposed by Google (or else by
the web-portal's coding?) Or is there a simple trick that I'm overlooking, to
get more than the 10-page limit?

Post a reply to this message

hi,

"Kenneth" <kdw### [at] gmailcom> wrote:
> ...
> I just did a search for 'Kenneth' (by itself) and see that Google presents 10
> pages of old entries MAXIMUM, with 10 hits per page. That's only 100 hits.

<tinfoil-hat>you're on the list of persons only allowed limited access to
reality</tinfoil-hat>  :-)

I see more than 600, see attached.


regards, jr.

Post a reply to this message

Attachments:
Download 'kw.png' (15 KB)

Preview of image 'kw.png'

Op 14-2-2023 om 15:38 schreef jr:
> hi,
> 
> "Kenneth" <kdw### [at] gmailcom> wrote:
>> ...
>> I just did a search for 'Kenneth' (by itself) and see that Google presents 10
>> pages of old entries MAXIMUM, with 10 hits per page. That's only 100 hits.
> 
> <tinfoil-hat>you're on the list of persons only allowed limited access to
> reality</tinfoil-hat>  :-)
> 
> I see more than 600, see attached.
> 

I got 670.

-- 
Thomas

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> hi,
>
> "Kenneth" <kdw### [at] gmailcom> wrote:
> > ...
> > I just did a search for 'Kenneth' (by itself) and see that Google presents 10
> > pages of old entries MAXIMUM, with 10 hits per page. That's only 100 hits.
>
> <tinfoil-hat>you're on the list of persons only allowed limited access to
> reality</tinfoil-hat>  :-)
>
> I see more than 600, see attached.
>

Ah, that's more like it! :-)

I see that my own search says "about 573 results". About?

But web portal/Google only presents 10 pages of the hits, sadly.

Post a reply to this message

Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> Getting google to properly index the site is a constant battle. I have done all they
say I should do and set everythi
ng up using their webmaster tools. All I can do is wait and hope that they catch up.
It's difficult in that google may 
say one thing but do another; this is part of the never-ending battle between search
engines and those who would wish t
o influence the results/come out at the top for a query.


"According to a report from the Associated Press, Google is stepping up its so

by indoctrinating people to disbelieve what tech big wigs consider to be


Yet another reason to run, not walk away from something rife with corruption.

Post a reply to this message

Hi(gh)!

On 14.02.23 15:38, jr wrote:

> <tinfoil-hat>you're on the list of persons only allowed limited access to
> reality</tinfoil-hat>  :-)

There *is* no such thing as "reality" at all! You altogether are nothing 
but tiny little conscious-entity routines within the framework of 
KHYBERSPACE! Face it!

See you there!

Yadgar

Post a reply to this message

On 14/02/2023 23:45, William F Pokorny wrote:
> pokorny:tmp$ telnet news.povray.org 119
> Trying 203.29.75.35...
> Connected to news.povray.org.
> Escape character is '^]'.
> 200 news.povray.org DNEWS Version  5.7b1,, S0, posting OK
> article <5e9cce19@news.povray.org>
> ... I get what looks like some large mime dump that I suspect is
> due some attached images... Probably the header is there, but not sure how to see
the header in this case.

You can use 'head <5e9cce19@news.povray.org>' instead of 'article' in this case. And
to get just the body minus the headers you can use 'body'.

> What to make of this? Some of my stuff is on the server and other stuff is not?

I can't say for sure what's there and what's not. To be totally sure I'd need to write
some code to cross-reference posts that are in the database against those in the news
server. Generally speaking if an article is removed from the news server by some
automated process (expire or disk usage) they will remain in the DB. If they are
explicitly removed (by the user or an admin) then they will be gone from both.

At the moment there's no way to specifically search the web view's DB (or more
specifically, no way unless google gets their shit together).

That said, I have improved the search pages by switching to google's paid search
service. The results on the free tier which I was using were pretty terrible.

However that doesn't fix the issue of missing pages or improper dates etc which we've
seen.

I have made another attempt to get google to digest the site by providing them with a
sitemap (tried this in the past and it didn't help, but worth trying again).

If you were really keen to see if a particular message is in the web DB you can grab
the sitemap and download the files it refers to, then grep for the message ID. The
sitemap is currently at https://news.povray.org/gsitemap.xml. Note this is (currently)
static, generated as of a few hours ago. If it works then I'll set up a cronjob to
keep it updated.

In addition to the above I also retain log files containing all content posted to the
server since 2004 (at least in theory - the earliest is 2004 and latest is today, but
I have yet to check that they all line up wrt start/end so there's no missing dates).
But accessing those shouldn't be necessary provided the web DB has everything.

If after checking the above-mentioned resources doesn't find a message you think is
missing it probably wouldn't be too difficult to throw together a simple perl script
to pull every post you made from the log files.

-- Chris

Post a reply to this message