POV-Ray : Newsgroups : povray.binaries.images : Denoising POV-Ray images in Blender Server Time
29 Mar 2024 03:05:18 EDT (-0400)
  Denoising POV-Ray images in Blender (Message 24 to 33 of 51)  
<<< Previous 10 Messages Goto Latest 10 Messages Next 10 Messages >>>
From: Bald Eagle
Subject: Re: Denoising POV-Ray images in Blender
Date: 3 Feb 2023 06:20:00
Message: <web.63dced4113ac33281f9dae3025979125@news.povray.org>
Thomas de Groot <tho### [at] degrootorg> wrote:

> Possibly the major crash POV-Ray experienced some time ago destroyed
> part of the index too? However, I agree with you: items are often
> difficult/impossible to find.

I think there are ways to get search engines to recrawl/reindex a site over time
- might be worth looking into.


Post a reply to this message

From: Cousin Ricky
Subject: Re: Denoising POV-Ray images in Blender
Date: 3 Feb 2023 08:27:47
Message: <63dd0bd3$1@news.povray.org>
On 2023-02-03 03:17 (-4), Thomas de Groot wrote:
> Op 03/02/2023 om 01:04 schreef Samuel B.:
> 
>> I do wish it was easier to search for old things, though. It seems
>> like all the
>> search engines only keep a partial index of the newsgroups.
>>
> Possibly the major crash POV-Ray experienced some time ago destroyed
> part of the index too? However, I agree with you: items are often
> difficult/impossible to find.

No, the searches skipped results even before the crash.  I can't count
the number of times I've had to hunt down an old post manually because
the search engine didn't find it.


Post a reply to this message

From: Bald Eagle
Subject: Re: Denoising POV-Ray images in Blender
Date: 3 Feb 2023 13:45:00
Message: <web.63dd555f13ac33281f9dae3025979125@news.povray.org>
Cousin Ricky <ric### [at] yahoocom> wrote:

> No, the searches skipped results even before the crash.  I can't count
> the number of times I've had to hunt down an old post manually because
> the search engine didn't find it.

There are instructions for "building your own search engine" - and i would
imagine this would be pretty manageable given it could be restricted to just
this site.

No idea who could set up a rudimentary test of this, but at the very least it
would crawl and index the entire site.

I think it would be really great to bea able to scroll through all of the zip,
inc, mcr, and other such files in a "digest" which was just the search
results...

Also, you could search by username, which would be particularly useful when
hunting down a file when you know who the author was.


Post a reply to this message

From: Chris Cason
Subject: Re: Denoising POV-Ray images in Blender
Date: 13 Feb 2023 12:02:19
Message: <63ea6d1b@news.povray.org>
On 4/02/2023 00:27, Cousin Ricky wrote:
> No, the searches skipped results even before the crash.  I can't count
> the number of times I've had to hunt down an old post manually because
> the search engine didn't find it.

Getting google to properly index the site is a constant battle. I have done all they
say I should do and set everything up using their webmaster tools. All I can do is
wait and hope that they catch up. It's difficult in that google may say one thing but
do another; this is part of the never-ending battle between search engines and those
who would wish to influence the results/come out at the top for a query.

I spent probably a full day working on this on and off using various of the
google-suggested methods and thought I had it sorted. But clearly if stuff is missing
then I need to re-visit it.

Screenshot of search console report is below: it is visiting us (and I do see them in
the server logs) but some things don't make sense (e.g. total bytes).

-- Chris


Post a reply to this message


Attachments:
Download 'crawlstats.png' (198 KB)

Preview of image 'crawlstats.png'
crawlstats.png


 

From: yesbird
Subject: Re: Denoising POV-Ray images in Blender
Date: 13 Feb 2023 12:30:00
Message: <web.63ea735413ac33283691b85710800fb2@news.povray.org>
Hi, Chris.

Working more than 8 years as an internet search system developer I can say that
SEO is a difficult way. Too much concurrency, no predictable results without
knowledge about the relevance function of the document body, which is a top
secret of Google.

At the same time there are some well known workarounds: SEO partnership, writing
posts on specialized forums, that are thematically related to our, for example.

Most reliable way is Google Ads, but it'is too expensive.

All the best,
--
YB


Post a reply to this message

From: Bald Eagle
Subject: Re: Denoising POV-Ray images in Blender
Date: 13 Feb 2023 13:50:00
Message: <web.63ea863713ac33281f9dae3025979125@news.povray.org>
Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> Getting google to properly index the site is a constant battle. I have done all they
say I should do and set everythi
ng up using their webmaster tools. All I can do is wait and hope that they catch up.
It's difficult in that google may 
say one thing but do another; this is part of the never-ending battle between search
engines and those who would wish t
o influence the results/come out at the top for a query.
>
> I spent probably a full day working on this on and off using various of the
google-suggested methods and thought I ha
d it sorted. But clearly if stuff is missing then I need to re-visit it.


I don't know anything about what software this newsgroup is running on or web
development in general,  but I've used other forums that were set up on Drupal
or Simple Machines, or other forum software like that - some by people who I
can't really say had very much tech expertise at all (I have no idea if they had
help) and those forums had "internal search" tools/plugins/features that didn't
seem to rely on an outside search engine to provide the results.

When looking for information on my own system, I've used grep to dig through
files and find little snippets of what I was looking for.

How did the old search engines work?   Lycos, WebCrawler, etc....

Can we get another search engine to index the site?   Brave browser search,
Yandex, ...?

I don't know if this forum software has features like that, that can be added
via a plugin or something else, or if there's another platform that everything
can be migrated to that does (I know, I know - you'd rather drink drano....)

Just wondering what the non-big-tech-search-engine options are.

-BW


Post a reply to this message

From: Chris Cason
Subject: Re: Denoising POV-Ray images in Blender
Date: 14 Feb 2023 02:21:20
Message: <63eb3670@news.povray.org>
On 4/02/2023 05:41, Bald Eagle wrote:
> There are instructions for "building your own search engine" - and i would
> imagine this would be pretty manageable given it could be restricted to just
> this site.
> 
> No idea who could set up a rudimentary test of this, but at the very least it
> would crawl and index the entire site.

There are some open-source search engines that can index the site, and I have
considered using one.

If I did it should solve the problem for local searches. However at the moment fixing
the google search results has higher priority since that's what a typical POV-Ray user
would use to ask about something. If it doesn't index all pages we would have cases
where a user searched for something and while the answer actually exists and is in our
groups, google hasn't included it, so they don't get a relevant result.

> I think it would be really great to bea able to scroll through all of the zip,
> inc, mcr, and other such files in a "digest" which was just the search
> results...
> 
> Also, you could search by username, which would be particularly useful when
> hunting down a file when you know who the author was.

Google provides means to understand what it's indexing. In particular, we can use
structured data as set out in
https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

All posts on the newsgroup have a JSON-LD object telling search engines that the
message is a forum post (type is DiscussionForumPosting) and other attributes,
specifically including the author.

If you are using the web view you can see this easily: view the source of this post in
your browser (usually control-U) then search for the word DiscussionForumPosting and
you will find them. For example here's what we said about your post:

<script type="application/ld+json">
{
          "@context": "https://schema.org",
          "@type": "DiscussionForumPosting",
          "@id": "#web.63dd555f13ac33281f9dae3025979125%40news.povray.org",
          "headline": "Re: Denoising POV-Ray images in Blender",
          "dateCreated": "2023-02-03T18:45:00+00:00",
          "datePublished": "2023-02-03T18:45:00+00:00",
          "author": {
                  "@type": "Person",
                  "name": "Bald Eagle"
          }
}
</script>

If I run the current page of this thread through the Google 'Rich Results Test' it
shows it saw and understood all the posts (the warnings are just for optional fields).
I have expanded the row where it saw your message (see attachment).

As far as I can tell the issue is Google isn't indexing *all* of our pages, regardless
of what I do (for example I tried using sitemaps in various forms and it still didn't
fetch all the URL's in the sitemap). I ran out of time trying to sort this and just
left it alone, hoping google would catch up eventually.


Post a reply to this message


Attachments:
Download 'richresults.png' (186 KB)

Preview of image 'richresults.png'
richresults.png


 

From: William F Pokorny
Subject: Re: Denoising POV-Ray images in Blender
Date: 14 Feb 2023 04:59:44
Message: <63eb5b90$1@news.povray.org>
On 2/14/23 02:18, Chris Cason wrote:
> As far as I can tell the issue is Google isn't indexing *all* of our 
> pages, regardless of what I do (for example I tried using sitemaps in 
> various forms and it still didn't fetch all the URL's in the sitemap). I 
> ran out of time trying to sort this and just left it alone, hoping 
> google would catch up eventually.

Hi Chris,

To chime in with a related topic; Many of us were using Thunderbird's 
search facilities of the newsgroups over the formally less reliable web 
based search / google search.

As, I think, you are aware, after the recovery from the crash the news 
server is maintaining much less of the overall posting history. It's 
often impossible to get to any older posts these days via Thunderbird.

I remember you saying the POV-Ray news server had been somewhat mangled 
in the past to keep posts in most, but not all forums, for a very long 
time. Something much longer the usual 12 months or so. My 'guess' is 
that when the news server was restored most, if not all, news server 
forums are now dropping posts after a much shorter time (12 months?).

Whether this worth fixing or not I'm unsure. Expect these days the 
newsgroup interface not the primary one for most users. Having a 
reliable way to search the forums should be primary the goal.

Aside: As I find time I've been manually downloading my own posts via 
the web interface - and some posts of others - to plain directories of 
per topic text for the thread and useful attachments. I can then use 
linux commands as mentioned by Bill W to find my forum 'notes.' I've 
been burning time since the crash doing this and I'm nowhere near 
complete. I wish these days I'd not relied so much on the newsgroups as 
a place to record stuff! :-)

Bill P.


Post a reply to this message

From: Chris Cason
Subject: Re: Denoising POV-Ray images in Blender
Date: 14 Feb 2023 06:26:08
Message: <63eb6fd0$1@news.povray.org>
On 14/02/2023 20:59, William F Pokorny wrote:
> To chime in with a related topic; Many of us were using Thunderbird's search
facilities of the newsgroups over the formally less reliable web based search / google
search.
> 
> As, I think, you are aware, after the recovery from the crash the news server is
maintaining much less of the overall posting history. It's often impossible to get to
any older posts these days via Thunderbird.
> 
> I remember you saying the POV-Ray news server had been somewhat mangled in the past
to keep posts in most, but not all forums, for a very long time. Something much longer
the usual 12 months or so. My 'guess' is that when the news server was restored most,
if not all, news server forums are now dropping posts after a much shorter time (12
months?).
> 
> Whether this worth fixing or not I'm unsure. Expect these days the newsgroup
interface not the primary one for most users. Having a reliable way to search the
forums should be primary the goal.

I don't think the articles are being deleted. I do suspect that it's difficult or
impossible to get the NNTP server to give a reliable view of what messages are
available.

I'm not sure if it's because I kept my old Thunderbird NNTP cache across the crash or
whether I was lucky and it did download all the headers from the new server before it
started playing up. As it is, though, thunderbird shows me messages in
povray.binaries.images going back to 2005. In povray.bugreports it shows messages
going back to 1998, though if I try to load them it says there's no such article. The
retained articles (where I was able to get it to load from the server) in this group
seem to start around September 2009.

In PBI I have messages going back to 2005 and they all seem to load when I do spot
checks.

To test that an article really is on the server and not just locally cached you can
easily check using telnet as long as you know the article ID. Example:
  
> # telnet news.povray.org 119
> Trying 203.29.75.35...
> Connected to news.
> Escape character is '^]'.
> 200 news.povray.org DNEWS Version  5.7b1,, S0, posting OK

Ask for for an old post where I know the article ID:

> article <429e54b3$1@news.povray.org>
> 220 58213 <429e54b3$1@news.povray.org> article retrieved - head and body follows
> From: "trulayne" <tru### [at] newrrcom>
> Newsgroups: povray.general
> Subject: Equally spaced objects along a spline curve
> Date: Wed, 1 Jun 2005 19:37:06 -0500
[SNIP]
> TIA
> Dennis
> .

Select PBI as the current group:

> group povray.binaries.images
> 211 74934 66649 146750 povray.binaries.images selected

This tells us the oldest message is article number 66649 and the newest is 146750
(this is the message I'm replying to).

Check what 66649 was:

> head 66649
> 221 66649 <3d6bd206$1@news.povray.org> article retrieved - head follows
> From: "Zeger Knaepen" <zeg### [at] studentkuleuvenacbe>
> Newsgroups: povray.binaries.images
> References: <3d6b4d0b@news.povray.org> <3d6baacc@news.povray.org>
<3d6bb72e@news.povray.org> <3d6bbf6a$1@news.povray.org>
> Subject: Re: Odd blur effect: squared spheres
> Date: Tue, 27 Aug 2002 21:24:35 +0200
[SNIP]
> Xref: news.povray.org povray.binaries.images:66649
> .

Now check the last message, 146750

> head 146750
> 221 146750 <63eb5b90$1@news.povray.org> article retrieved - head follows
> Date: Tue, 14 Feb 2023 04:59:44 -0500
> MIME-Version: 1.0
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
>  Thunderbird/102.7.1
> Subject: Re: Denoising POV-Ray images in Blender
> Newsgroups: povray.binaries.images
> References: <63dd0bd3$1@news.povray.org>
>  <web.63dd555f13ac33281f9dae3025979125@news.povray.org>
>  <63eb3670@news.povray.org>
> Content-Language: en-US
> From: William F Pokorny <ano### [at] anonymousorg>
> In-Reply-To: <63eb3670@news.povray.org>
[SNIP]
> Xref: news.povray.org povray.binaries.images:146750
> .
> quit
> 205 closing connection - goodbye!
> Connection closed by foreign host.

AFAICT all the messages are still in the news server but perhaps it's not presenting
that to thunderbird in a way that it understands, at least in some cases?

Even if they were being deleted, they aren't lost forever as I have a backup with all
the posts prior to the crash and also each message is stored in the SQL db used for
the web view. Being able to *find* them is another issue, as google aren't being all
to helpful.

-- Chris


Post a reply to this message

From: Bald Eagle
Subject: Re: Denoising POV-Ray images in Blender
Date: 14 Feb 2023 07:20:00
Message: <web.63eb7c1713ac33281f9dae3025979125@news.povray.org>
Chris Cason <del### [at] deletethistoopovrayorg> wrote:

> If I did it should solve the problem for local searches. However at the moment
fixing the google search results has h
igher priority since that's what a typical POV-Ray user would use to ask about
something. If it doesn't index all pages
 we would have cases where a user searched for something and while the answer actually
exists and is in our groups, goo
gle hasn't included it, so they don't get a relevant result.

.....

> As far as I can tell the issue is Google isn't indexing *all* of our pages,
regardless of what I do (for example I tr
ied using sitemaps in various forms and it still didn't fetch all the URL's in the
sitemap). I ran out of time trying t
o sort this and just left it alone, hoping google would catch up eventually.

I would take a poll to confirm your assertion about typical methods and thus
priority.

I don't trust Google or any of the other Big Tech companies _at all_ anymore.
For years, I have experienced finding a result and then looking for the exact
same thing the very next day to share the link, and it doesn't come up in the
results AT ALL.  The search results are constantly changing, getting filtered,
and being manipulated.

If you make a reliable search feature, people would use that and then that would
de facto become the typical method for finding relevant posts - in a timely
manner.   Either way, it's a self-fulfilling argument.


Post a reply to this message

<<< Previous 10 Messages Goto Latest 10 Messages Next 10 Messages >>>

Copyright 2003-2023 Persistence of Vision Raytracer Pty. Ltd.