I am looking for a set of scenes that will test all (or at least most) of
the features of the official version of POV.  I have another important
requirement, though: they must render very quickly.

I want to be able to render all of these before each release of MegaPov to
make sure nothing has been broken.  Regression test scenes for MegaPov
features would also be useful.  Does anybody want to work on this?  Please
don't send individual scenes, as I do not have time to compile the set.  But
if somebody wants to create a set of scenes like this (using some existing
scenes from the POV distribution would be OK with me), I'd greatly
appreciate it.

I plan to use this for the development of MegaPov (I am currently speaking
only for myself, not the POV-Team), but regression testing will also be
useful in development of the official version of POV, so the POV-Team may
wish to also use this set of scenes.

-Nathan

Post a reply to this message

On Mon, 10 Jan 2000 13:43:33 -0500, Nathan Kopp wrote:
>I want to be able to render all of these before each release of MegaPov to
>make sure nothing has been broken.  Regression test scenes for MegaPov
>features would also be useful.  Does anybody want to work on this?  Please
>don't send individual scenes, as I do not have time to compile the set.  But
>if somebody wants to create a set of scenes like this (using some existing
>scenes from the POV distribution would be OK with me), I'd greatly
>appreciate it.

How do you propose to determine whether a given test was successful?  Visual
inspection, or some kind of comparison to a standard render?

-- 
These are my opinions.  I do NOT speak for the POV-Team.
The superpatch: http://www2.fwi.com/~parkerr/superpatch/
My other stuff: http://www2.fwi.com/~parkerr/traces.html

Post a reply to this message

Nathan Kopp wrote:
> 
> I am looking for a set of scenes that will test all (or at least most) of
> the features of the official version of POV.  I have another important
> requirement, though: they must render very quickly.
> 
> I want to be able to render all of these before each release of MegaPov to
> make sure nothing has been broken.  Regression test scenes for MegaPov
> features would also be useful.  Does anybody want to work on this?  Please
> don't send individual scenes, as I do not have time to compile the set.  But
> if somebody wants to create a set of scenes like this (using some existing
> scenes from the POV distribution would be OK with me), I'd greatly
> appreciate it.
> 
> I plan to use this for the development of MegaPov (I am currently speaking
> only for myself, not the POV-Team), but regression testing will also be
> useful in development of the official version of POV, so the POV-Team may
> wish to also use this set of scenes.
> 
> -Nathan

Perhaps I am missing something here Nathan but there are already example
scenes provided with the official distribution that cover all of it's
features and the majority of these do render quite fast. Perhaps what
you are asking for is scenes the cover several of the features in one
multifunctional file ?

I also pose the same questions that Ron raised concerning how you will
evaluate the results. The more features you illustrate in a single
image the harder it is to evaluate minor programming problems because
scale has a tendancy to minimize visible defects. Single feature scenes
generaly have the example shown up close in detail increasing the chance
of catching a defect. You might get around this by rendering at higher
resolutions but the defeats your need for fast rendering times.

What is regression testing ?

-- 
Ken Tyler -  1300+ Povray, Graphics, 3D Rendering, and Raytracing Links:
http://home.pacbell.net/tylereng/index.html http://www.povray.org/links/

Post a reply to this message

> Perhaps I am missing something here Nathan but there are already example
> scenes provided with the official distribution that cover all of it's
> features and the majority of these do render quite fast. Perhaps what
> you are asking for is scenes the cover several of the features in one
> multifunctional file ?

I've been using various modified versions of allscene.sh (in the Unix
versions) to do this.  I use effectively the same script for MegaPov. 
At some point I'll probably add a bunch of scenes from the IRTC CD's to
those I test, as well as some that were released in previous versions. 
I've also considered that such a large number of scenes of medium
resolution might be useful as a replacement for the somewhat dated
povbench benchmark.

> I also pose the same questions that Ron raised concerning how you will
> evaluate the results. The more features you illustrate in a single
> image the harder it is to evaluate minor programming problems because
> scale has a tendancy to minimize visible defects. Single feature scenes
> generaly have the example shown up close in detail increasing the chance
> of catching a defect. You might get around this by rendering at higher
> resolutions but the defeats your need for fast rendering times.

There are two ways I do this: rendering a bunch of stuff with low
resolution just to see whether it works (I've known the render to fail,
and I've known the render to hang).  Running allscene.sh on my machine
typically takes around 25 minutes.  Other methods involve rendering at
larger resolution (over a period of days) and visually eyeballing for
unusual artifacts.  I've seen all the scenes in the standard
distribution enough to have some idea if something goes completely
screwy, and I also have "canonical" high-resolution versions saved on my
hard drive.  The MegaPov example scenes are not so familiar.

> What is regression testing ?

Running through the full set of tests for the sake of thoroughness,
sometimes defined as having a special emphasis on those tests that
previously revealed bugs.

-Mark Gordon

Post a reply to this message

"Nathan Kopp" <Nat### [at] Koppcom> writes:

> I am looking for a set of scenes that will test all (or at least most) of
> the features of the official version of POV.  

It is a good thing to create such a test set. Keep it going! For automatic 
testing of the result you may use Warps program that calculates the 
difference of two images.

Thomas

-- 
http://thomas.willhalm.de/ (includes pgp key)

Post a reply to this message

On Mon, 10 Jan 2000 22:25:48 -0600, Mark Gordon wrote:
>There are two ways I do this: rendering a bunch of stuff with low
>resolution just to see whether it works (I've known the render to fail,
>and I've known the render to hang).  Running allscene.sh on my machine
>typically takes around 25 minutes.  Other methods involve rendering at
>larger resolution (over a period of days) and visually eyeballing for
>unusual artifacts.  I've seen all the scenes in the standard
>distribution enough to have some idea if something goes completely
>screwy, and I also have "canonical" high-resolution versions saved on my
>hard drive.  The MegaPov example scenes are not so familiar.

The reason I asked is because it's something I've thought about myself.
I think it should be relatively easy to work out an automated solution 
for the cases where there aren't any failures:  

Use pnminvert to invert the canonical copy of the image, then use ppmmix 
to mix 50% of it with 50% of the test image.  Finally, run ppmhist on
the result and examine its output to see how many pixels are significantly 
different from solid 50% gray.  If it's over a certain threshold, you can 
guarantee there's been a significant change in how the image was rendered.

Adjusting the threshold and the definition of "significantly different from
solid 50% gray" allows you to detect smaller and smaller differences in
rendering, up to and including perfect reproduction of the original image.
So you can sort the images into categories: "really different - probably a bug",
"A little different", and "perfect".
-- 
These are my opinions.  I do NOT speak for the POV-Team.
The superpatch: http://www2.fwi.com/~parkerr/superpatch/
My other stuff: http://www2.fwi.com/~parkerr/traces.html

Post a reply to this message

This is what I was thinking.

-Nathan

Ron Parker <ron### [at] povrayorg> wrote...
> The reason I asked is because it's something I've thought about myself.
> I think it should be relatively easy to work out an automated solution
> for the cases where there aren't any failures:
>
> Use pnminvert to invert the canonical copy of the image, then use ppmmix
> to mix 50% of it with 50% of the test image.  Finally, run ppmhist on
> the result and examine its output to see how many pixels are significantly
> different from solid 50% gray.  If it's over a certain threshold, you can
> guarantee there's been a significant change in how the image was rendered.

Post a reply to this message

Ken <tyl### [at] pacbellnet> wrote...
>
> Perhaps I am missing something here Nathan but there are already example
> scenes provided with the official distribution that cover all of it's
> features and the majority of these do render quite fast. Perhaps what
> you are asking for is scenes the cover several of the features in one
> multifunctional file ?

Do they cover all of the features?  I know they cover many, but I don't know
if they get everything and they way everything interacts (such as reflection
used within a texture_map, which is the source of a bug in MegaPov 0.3).

I guess that for now I will use those and consider it good enough.  :-)

-Nathan

Post a reply to this message

Nathan Kopp wrote:

> Do they cover all of the features?

As far as I know yes. There might be some i/o directives not covered
and all of the possible string examples you can come up with are not
present but the basics are all covered to the best of my knowlege.

>  I know they cover many, but I don't know
> if they get everything and they way everything interacts (such as reflection
> used within a texture_map, which is the source of a bug in MegaPov 0.3).

Well that is going to be a problem no matter how hard you try to design a
comprehensive suite of tests. This program has so many diverse features
that the possible combinations are almost limitless.
 
> I guess that for now I will use those and consider it good enough.  :-)

Works for me :)

-- 
Ken Tyler -  1300+ Povray, Graphics, 3D Rendering, and Raytracing Links:
http://home.pacbell.net/tylereng/index.html http://www.povray.org/links/

Post a reply to this message

You could also use Paint Shop Pro to do a difference on the two scenes (the
master and the test) and inspect the result.

Jim

--

Ron Parker <ron### [at] povrayorg> wrote in message
news:387b302d@news.povray.org...
> On Mon, 10 Jan 2000 22:25:48 -0600, Mark Gordon wrote:
> >There are two ways I do this: rendering a bunch of stuff with low
> >resolution just to see whether it works (I've known the render to fail,
> >and I've known the render to hang).  Running allscene.sh on my machine
> >typically takes around 25 minutes.  Other methods involve rendering at
> >larger resolution (over a period of days) and visually eyeballing for
> >unusual artifacts.  I've seen all the scenes in the standard
> >distribution enough to have some idea if something goes completely
> >screwy, and I also have "canonical" high-resolution versions saved on my
> >hard drive.  The MegaPov example scenes are not so familiar.
>
> The reason I asked is because it's something I've thought about myself.
> I think it should be relatively easy to work out an automated solution
> for the cases where there aren't any failures:
>
> Use pnminvert to invert the canonical copy of the image, then use ppmmix
> to mix 50% of it with 50% of the test image.  Finally, run ppmhist on
> the result and examine its output to see how many pixels are significantly
> different from solid 50% gray.  If it's over a certain threshold, you can
> guarantee there's been a significant change in how the image was rendered.
>
> Adjusting the threshold and the definition of "significantly different
from
> solid 50% gray" allows you to detect smaller and smaller differences in
> rendering, up to and including perfect reproduction of the original image.
> So you can sort the images into categories: "really different - probably a
bug",
> "A little different", and "perfect".
> --
> These are my opinions.  I do NOT speak for the POV-Team.
> The superpatch: http://www2.fwi.com/~parkerr/superpatch/
> My other stuff: http://www2.fwi.com/~parkerr/traces.html

Post a reply to this message