On 06/13/2017 07:56 AM, Bald Eagle wrote:
> I was dabbling with PoseRay output files for a satellite, and POV-Ray choked on
> the "-A" option.
> 
> All was well when I commented that line out.
> 
> Considering how there are things that "ought" to work, and sometimes get missed
> or inadvertently broken, I was wondering if one of the things that ought to be
> considered, is some sort of "validation suite" - a set of standard files that
> will run through a long, as-complete-as-possible list of all past known bugs,
> and as many features, functions, and combinations of things as possible to
> automagically check to see if everything works.
> 
> This could have multiple uses, as it could serve as a repository for documenting
> 'bugs' and a checklist for making sure they are addressed, not forgotten, and
> remain fixed, documenting code examples, demonstrating to new users (and more
> experienced users unfamiliar with certain features) HOW and to a certain degree
> why - certain things in POV-Ray happen the way they do.
> (I'm thinking about utf-8, coincident surfaces, nearly coincident surfaces,
> sphere sweep and isosurface artifacts, .ini and command line options, etc.)
> 
> It would save work in development testing, as whatever was in the file(s) could
> just stay there, and then only new things would have to be added.
> A large amount of debugging could be accomplished with a simple 'open file' and
> 'run'.
> 
> Just an idea.
> 
> 

With respect to the -A in an ini file causing issues would you please 
post the exact line or perhaps the files needed to reproduce. Just 
adding -A to an ini file here is not failing in my 3.7.1 release branch 
(or 3.7.0)

I am on Ubuntu linux and the options parsing code is not completely 
shared between windows and linux at this time - so perhaps you are 
seeing a windows only issue.

---

The "validation suite" is a valid :-) idea, but one which pretty quickly 
needs a "validation team," "validation computers" and such to really 
keep up with it. Such testing is a tedious, never ending, job so finding 
volunteers that will really stick with such testing for the long haul is 
not easy.

Each of us contributing tends to run the shipped demo scenes. I have 
some hacks at my own testing framework that will do "sanity runs" of all 
400 or so scenes automatically checking return codes, but not much more. 
It takes a while to turn on my little two core, so I don't run all 400 
very often.

I've also got some other smaller test sets for include files and parser 
issues that I've slowly been building up.

Using my parser testing as an example - my current template for parser 
tests looks something like this:

//
//             +- Expected return code.
//             | +- Expected message, if any.
//             | |                                    +- Optional >=16 chars
//             | |                                    | of git hash where
//             | |                                    | in-version change.
// EXPECT 3.70 0                                    ;
// EXPECT 3.71 1 Parse Error: No normal type given. ;
// EXPECT 3.72 1 Parse Error: No normal type given. ;
#default {
     pigment { rgb <1,1,1> }
     normal { bump_size 1.5 }             // <--- Fail
     finish { phong 0.8 phong_size 200 }
}


// <file><language_version_if_matters>.in names have prefixes:
//   F_or_P_    - Where pass or fail changes over time.
//   Fail_      - Expect to always fail no matter version.
//   Pass_      - Expect to always pass no matter version.
//   Warning_   - Expect to generate a warning message 'runtests' will 
verify.
//   Debug_     - Expect to generate a debug message 'runtests' will verify.
//   Error_     - Expect to fail due SDL #error with message 'runtests' 
will verify.

So, for example, I have these files for the somewhat recent changes to 
lathe spline parser behavior:

Fail_LatheCubic1n2NegX.inc
Fail_LatheCubic2n3NegX.inc
Fail_LatheCubic2ndNegX.inc
Fail_LatheCubic3n4NegX.inc
Fail_LatheCubicAllNegX.inc
Fail_LatheLinear1n2NegX.inc
Fail_LatheLinear2n3NegX.inc
Fail_LatheLinear2ndNegX.inc
Fail_LatheLinear3n4NegX.inc
Fail_LatheLinearAllNegX.inc
Fail_LatheQuadratic1n2NegX.inc
Fail_LatheQuadratic2n3NegX.inc
Fail_LatheQuadratic2ndNegX.inc
Fail_LatheQuadratic3n4NegX.inc
Fail_LatheQuadraticAllNegX.inc
F_or_P_LatheBezier1stAnd2ndControlNegX370.inc
F_or_P_LatheBezier1stAnd2ndControlNegX371.inc
F_or_P_LatheBezier1stControlNegX370.inc
F_or_P_LatheBezier1stControlNegX371.inc
F_or_P_LatheBezier2ndControlNegX370.inc
F_or_P_LatheBezier2ndControlNegX371.inc
F_or_P_LatheBezierPointFirstNegX370.inc
F_or_P_LatheBezierPointFirstNegX371.inc
F_or_P_LatheBezierPointLastNegX370.inc
F_or_P_LatheBezierPointLastNegX371.inc
F_or_P_LatheLinear1stNegX370.inc
F_or_P_LatheLinear1stNegX371.inc
F_or_P_LatheLinearLastNegX370.inc
F_or_P_LatheLinearLastNegX371.inc
F_or_P_LatheQuadraticLastNegX370.inc
F_or_P_LatheQuadraticLastNegX371.inc
Pass_LatheBezier.inc
Pass_LatheCubic.inc
Pass_LatheLinear.inc
Pass_LatheQuadratic.inc

When someone recently drove the change to make certain invalid spline 
specifications warnings and not errors where the language version was 
<=3.7, I had to go back and add additional 370 and 371 specific files 
and track the commit in the 370 ones where the behavior changes over time.

It gets to be a real pain to keep this stuff up and I'm honestly not 
sure I'll continue the effort... 
 


Bill P.

Post a reply to this message