OK so I'll start to address some of the things needed to be done to get
a good, solid 4.8 release. I'll spread this across several messages as I
get time.

(Before I begin: I have yet to fully catch up with this thread as there
are a number of messages I've left unread - I'll catch up later,
apologies if I duplicate/repeat something already covered).

One of the main issues we face (and where we could really do with
community help as this is hard to automate) is making sure all the
combined changes firstly don't outright break any of the shipped scenes
and snippets (by that I mean cause parse/render to fail with an error or
exception), and secondly don't cause any unexpected *visual* artifacts
(or at least not any unexpected ones that would fail the 'principle of
least astonishment' test that I ramble on about occasionally).

The first can be automated to a reasonable extent. The latter is really
part of the former (as to view the image you need to complete the
render) and is really, really difficult to automate. The reason for that
is it's a matter of perception; of a human's ability to look at
something and see it for what it is rather than a collection of pixels.

Computers are OK at looking a collections of pixels mathematically if
they have a base set of images (rendered e.g. by v3.7) to compare
against. They can see if a pixel is different by 0.001% and can even be
taught to understand things like gamma changes but it's much more
difficult to get an algorithm that correctly interprets what a *user*
would consider a 'right' or 'wrong' change from an image as a whole.

It really needs user input. If we could get a cadre of all the above-
mentioned images rendered with 3.7 (plus a selection of good external
candidates like HOF images and some newer works), then the same with
3.8, and have them all put up side-by-side on a website or some other
means of allowing multiple collaborators to view, rate and/or comment on
the differences then it would be a large step forward to getting 3.8 out
of the door.

-- Chris

Post a reply to this message

hi,

Chris Cason <del### [at] deletethistoopovrayorg> wrote:
> ...
> It really needs user input. If we could get a cadre of all the above-
> mentioned images rendered with 3.7 (plus a selection of good external
> candidates like HOF images and some newer works), then the same with
> 3.8, and have them all put up side-by-side on a website or some other
> means of allowing multiple collaborators to view, rate and/or comment on
> the differences then it would be a large step forward to getting 3.8 out
> of the door.

I have versions 3.7.0.8 and 3.8.0-alpha.10064268 installed on a machine, and
will be happy to render scenes with both and post the images.  perhaps you could
start a new thread just for posting results? (until the "website or some other
means" becomes available)

there are around 400 scene files in v3.7; how to ensure that users do not
duplicate work?  other than that, I can make a start tomorrow.

btw, regarding side-by-side, Linux users will have the (ImageMagick) 'montage'
utility which could be useful here.  a command like:

  $ montage v37img.png v38img.png -tile 1x2 -geometry +0+0 imgcmp.png

creates a single image with the v38 image below the v37; without internal
border(s).


regards, jr.

Post a reply to this message

On Tue, 08 Sep 2020 04:55:43 +1000, Chris Cason wrote:

> It really needs user input. If we could get a cadre of all the above-
> mentioned images rendered with 3.7 (plus a selection of good external
> candidates like HOF images and some newer works), then the same with
> 3.8,
> and have them all put up side-by-side on a website or some other means
> of allowing multiple collaborators to view, rate and/or comment on the
> differences then it would be a large step forward to getting 3.8 out of
> the door.

Building on jr's suggestion, I wonder if the builds are happening with 
some sort of CI/CD system, if it might be possible to automate the 
montage and include a 'diff' image in it to show areas where the 
algorithms differ.  That might make it easier for someone looking at the 
results to see what the differences are (and to look past areas that 
might look different but are visually identical - because human eyes are 
weird sometimes).



-- 
"I learned long ago, never to wrestle with a pig. You get dirty, and 
besides, the pig likes it." - George Bernard Shaw

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:

> I have versions 3.7.0.8 and 3.8.0-alpha.10064268 installed on a machine, and
> will be happy to render scenes with both and post the images.  perhaps you could
> start a new thread just for posting results? (until the "website or some other
> means" becomes available)

I use https://postimages.org/   (also https://postimg.cc/)
https://postimg.cc/ZBg50sLh

> there are around 400 scene files in v3.7; how to ensure that users do not
> duplicate work?  other than that, I can make a start tomorrow.

:O  I didn't realize there were that many!

IIRC there's an .ini file in the distro to render all of the images for the
insert menu.   So there's a starting list, and a post-render command could be
added to do the side-by-side - and the diff.

> btw, regarding side-by-side, Linux users will have the (ImageMagick) 'montage'
> utility which could be useful here.  a command like:
>
>   $ montage v37img.png v38img.png -tile 1x2 -geometry +0+0 imgcmp.png
>
> creates a single image with the v38 image below the v37; without internal
> border(s).

Very cool, It would be great to have a small file with useful techniques like
this in a future distro.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> Chris Cason <del### [at] deletethistoopovrayorg> wrote:
> > ...
> > It really needs user input. If we could get a cadre of all the above-
> > mentioned images rendered with 3.7 (plus a selection of good external
> > candidates like HOF images and some newer works), then the same with
> > 3.8, and have them all put up side-by-side on a website or some other
> > means of allowing multiple collaborators to view, rate and/or comment on
> > the differences then it would be a large step forward to getting 3.8 out
> > of the door.
> ...
> there are around 400 scene files in v3.7; ...

most of the distribution scene files have one or more "recommended" render
settings in their header which should be used.  so I've hacked up a set of
scripts (for GNU/Linux) to do the rendering with two POV-Ray versions and create
side-by-side-compare images, to explore, and perhaps build on.

attached archive has the scripts and a small 'readme' which outlines the steps.
note I forgot to mention where to edit/change the POV-Ray executable names,
which may/will need adapting; 'mkcmpimgs.tcl' line 11.

Jim Henderson:
could the above be useful in the context/system you're thinking of?


regards, jr.

Post a reply to this message

Attachments:
Download 'mkcmpimg.tar.gz' (4 KB)

"jr" <cre### [at] gmailcom> wrote:
> ...
> most of the distribution scene files have one or more "recommended" render
> settings in their header which should be used.  so I've hacked up a set of
> scripts (for GNU/Linux) to do the rendering with two POV-Ray versions and create
> side-by-side-compare images, to explore, and perhaps build on.

and after I posted this, I cleaned out my '/tmp/' directory and followed the
instructions in the 'readme.txt', and ran into trouble.  </sigh>
early in the "driver" script's development, when I had the basic functionality,
I commented out the commands and replaced them with to-screen output of the
command-lines otherwise executed; the script in the archive is (deliberately)
shipped in that "crippled" state.

the first snag was (probably) of my own making, an error occurred when running
'addscenes.sh', see attached for details.

and then there are a few scene files which, for one reason or other, need to be
excluded from the run; four updates to the database required, details attached.

two scenes actually fail for v3.8, but work for 3.7.  again, see attached.


regards, jr.

Post a reply to this message

Attachments:
Download 'cmpimg.txt' (2 KB)

On 9/11/20 9:29 PM, jr wrote:
> "jr" <cre### [at] gmailcom> wrote:
...
> 
> and then there are a few scene files which, for one reason or other, need to be
> excluded from the run; four updates to the database required, details attached.
> 
> two scenes actually fail for v3.8, but work for 3.7.  again, see attached.
> 

We should re-start this testing discussion in a group other than off 
topic where the posts go poof after a time... (povray.beta-test ?)

When you run, are you running v3.7 shipped scenes with v3.7 povray and 
v3.8 scenes with v3.8 povray? Some scenes are different v3.7 to v3.8 to 
fix issues or align with v3.8 changes.

There are also new scenes in v3.8 where no comparison is possible. With 
these results should at least be run and looked over.

The allscene.sh script as I recall was not handling even all the v3.7 
base features and scenes, but been years since I was doing this sort of 
testing (when we pushed and dropped a v3.71).

When I run comparisons I almost always run with one thread despite this 
being slow. Otherwise some results - radiostiy involved ones for example 
- can be quite different, same version and file, run to run.

There is also the issues of all the jitter settings if you want to do 
detailed automatic comparisons - these jitters all need to be off in 
such cases. AA, area lights... Oh and you have to avoid some features 
like crand (subsurface?).

A kind of testing I don't believe has been done for v3.8 (even in the 
v3.71 testing) - except maybe by Christoph - is to render to all the 
output file formats.

Some of the outputs need a viewer (preferably not POV-Ray itself) which 
supports dithering and the higher bit depth outputs. Test here I guess 
would be do things look more or less OK output format to output format. 
We wouldn't need to run all the scenes for this testing. We might lean 
on one of the image packages to do conversions and comparisons, but 
running such programs can get pretty detailed if the outputs are not all 
linear.

Further, we should read into bump_map, image_maps all the supported 
formats (tiff for example we can read, but not write).

There was a fair bit of file io work, fixes for bugs, better support for 
some formats in v3.8.

An example of, do 'new' v3.8 things still work, testing: Do all the new 
output dithering modes still work as they should. A concern I have is 
the time (6-7 years) since some of the earlier v3.8 changes were made 
and tested. Changes since the older changes have the potential to break 
or change (for better or worse) the earlier v3.8 changes. Without 
dedicated tests for the features, which we often dont' have, we won't 
know. Suppose a lot of these fringe features for mainstream users. Maybe 
this sort of testing not at the top of the list...

Anyway. Please forgive the typos/breaks in thought. Rushing ahead of my 
first coffee.

---
IIRC with respect to those two scenes which are not running for you. The 
first is a real error and I thought the scene itself was fixed in v3.8?

With the second there was an old include file with a vertical feed or 
form feed character in it which died with Christoph's new parser 
changes. My argument was to just fix the shipped file as probably people 
should avoid such characters these days. His argument was there might be 
old files in the wild with those characters and the parser should handle 
them. What I don't know is whether he made those parser updates...

My last question is whether you are running the last v3.8 official 
pre-release or the master branch at last commit?

There were fixes and other changes in the commits between the last 
official pre-release and what is in master today. Round about way to say 
we need to decide from what "commit" we are going to try and release. I 
think many of us are running code based off the last commit (Jerome, me, 
Dick?), but maybe most are running - can only run - the last pre-release?

Bill P.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 9/11/20 9:29 PM, jr wrote:
> > "jr" <cre### [at] gmailcom> wrote:
> ...
> We should re-start this testing discussion in a group other than off
> topic where the posts go poof after a time... (povray.beta-test ?)

done.  see 'v3.7 v3.8 image compare'.

Post a reply to this message

On 8/9/2020 1:57 PM, jr wrote:

> 
> great, there's news in the news section, and the HOF images change.  (it feels
> ... less abandoned.  thanks)
> 
> 
> regards, jr.
> 

Hmm, it occurs to me I should have asked to have qtpovray added to the news.

And I'm bummed that most of this thread rolled off before I could read it.

-- 
dik
Rendered 50,081,587,200 of 50,081,587,200 pixels (100%)

Post a reply to this message

hi,

Dick Balaska <dic### [at] buckosoftcom> wrote:
> On 8/9/2020 1:57 PM, jr wrote:
> > great, there's news in the news section, and the HOF images change.  (it feels
> > ... less abandoned.  thanks)
> >
>
> Hmm, it occurs to me I should have asked to have qtpovray added to the news.

good luck with that undertaking.

> And I'm bummed that most of this thread rolled off before I could read it.

from the web i/f, I see the thread complete/intact.


regards, jr.

Post a reply to this message