Hello / Hi
I have installed PovRay 3.8.0-x.10064738.unofficial last sunday on Ubuntu with
the sources compilation.
To easily switch between several povray versions I have the following files in
'usr/local/bin' :
'povray3-7' which is the binary of the stable 3.7 branch
'povray3-8' which is the binary of the 3.8.0[etc] described above
and a file 'povray' which is a symbolic link that can point to one of the two
binaries above.

I have a scene that renders with povray 3.7 but doesn't with v3.8.
With the 3.8 version, povray outputs the following message in the console:

==== [Parsing...] ==========================================================
File 'trees/kapok.tree' line 4: Possible Parse Error: Uncategorized error thrown
 at parser/parsertypes.cpp line 62.
Fatal error in parser: Uncategorized error.
Render failed

And I don't know how to solve this problem, even after parsing by myself
'kapok.tree' file.

Here is the output of the frist lines of 'kapok.tree' :
// TREE_NAME=Defaut;
//ROOT
#declare ROOTS=4;
#declare ROOT_RANDOM=13;
#declare FIRST_ROOT_AT_TRUNK=70;
#declare ROOT_LENGTH=200;
#declare ROOT_THICKNESS_CHANGE=0.85;

-----------------
In the pov file I have this line that loads 'kapok.tree' :
LoadTree("trees/kapok.tree")

And the 'LoadTree()' macro is defined like that :
#macro LoadTree(filePath)
 #include filePath
 #include "povtree/TOMTREE-1.5.inc"
#end

---------------
When I input 'povray --version' in the console, I have the following output:

POV-Ray 3.8.0-x.10064738.unofficial

This is an unofficial version compiled by:
 Antoine FAURE <https://www.ant01.fr>
 The POV-Ray Team is not responsible for supporting this version.

Copyright 1991-2019 Persistence of Vision Raytracer Pty. Ltd.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Built-in features:
  I/O restrictions:          enabled
  X Window display:          enabled (using SDL)
  Supported image formats:   gif tga iff ppm pgm hdr png jpeg tiff openexr
  Unsupported image formats: -

Compilation settings:
  Build architecture:  x86_64-pc-linux-gnu
  Built/Optimized for: x86_64-pc-linux-gnu (using -march=native)
  Compiler vendor:     gnu
  Compiler version:    g++-8 8
  Compiler flags:      -pipe -Wno-multichar -Wno-write-strings
-fno-enforce-eh-specs -Wno-non-template-friend -Wsuggest-override -s -O3
-ffast-math -march=native -pthread

In the hope that helps...

Post a reply to this message

On 3/19/19 4:01 PM, Warren wrote:
> Hello / Hi
> I have installed PovRay 3.8.0-x.10064738.unofficial last sunday on Ubuntu with
> the sources compilation.
> To easily switch between several povray versions I have the following files in
> 'usr/local/bin' :
> 'povray3-7' which is the binary of the stable 3.7 branch
> 'povray3-8' which is the binary of the 3.8.0[etc] described above
> and a file 'povray' which is a symbolic link that can point to one of the two
> binaries above.
> 
> I have a scene that renders with povray 3.7 but doesn't with v3.8.
> With the 3.8 version, povray outputs the following message in the console:
> 
...
> 

I can confirm the v38 issue. Also running Ubuntu 18.04 though g++7 vs 
Warren's 8.

It seems the v3.8 branch (commit 74b3ebe) needs a space ahead of the 
macro #end statement to work as follows:

#macro LoadThing(filePath)
     #include filePath
  #end                 // <-- Add extra space before #end and OK

LoadThing("zoomBangCrash.inc")

and whenever the include file has an empty line, C++ comment style line, 
C++ comment line and then a declare line. The following for 
zoomBangCrash.inc, less the line numbers, is also a v3.8 parse issue if 
there is no space ahead of the main file macro #end:

01
02 //
03 //
04 #declare A=1;

Perturb the include much and things work whether there is a space before 
the macro #end or not. Change the #declare A=1; to #declare Sphere00 = 
sphere { 0, 1 } for example and things are fine.

Bill P.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/19/19 4:01 PM, Warren wrote:
> > Hello / Hi
> > I have installed PovRay 3.8.0-x.10064738.unofficial last sunday on Ubuntu with
> > the sources compilation.
> > To easily switch between several povray versions I have the following files in
> > 'usr/local/bin' :
> > 'povray3-7' which is the binary of the stable 3.7 branch
> > 'povray3-8' which is the binary of the 3.8.0[etc] described above
> > and a file 'povray' which is a symbolic link that can point to one of the two
> > binaries above.
> >
> > I have a scene that renders with povray 3.7 but doesn't with v3.8.
> > With the 3.8 version, povray outputs the following message in the console:
> >
> ...
> >
>
> I can confirm the v38 issue. Also running Ubuntu 18.04 though g++7 vs
> Warren's 8.
>
> It seems the v3.8 branch (commit 74b3ebe) needs a space ahead of the
> macro #end statement to work as follows:
>
> #macro LoadThing(filePath)
>      #include filePath
>   #end                 // <-- Add extra space before #end and OK
>
> LoadThing("zoomBangCrash.inc")
>
> and whenever the include file has an empty line, C++ comment style line,
> C++ comment line and then a declare line. The following for
> zoomBangCrash.inc, less the line numbers, is also a v3.8 parse issue if
> there is no space ahead of the main file macro #end:
>
> 01
> 02 //
> 03 //
> 04 #declare A=1;
>
> Perturb the include much and things work whether there is a space before
> the macro #end or not. Change the #declare A=1; to #declare Sphere00 =
> sphere { 0, 1 } for example and things are fine.
>
> Bill P.

Hello / Hi
I made the changes you suggested and everything works fine now (I get all the
textures I normally have when I use v3.7 but with the v3.8 this time). In fact,
I use the v3.8 because there's a bug in the v3.7 with the conic_sweep prisms
when the camera is orthographic and its direction perpendicular to the caps of
the pyramid ( I posted a message about this bug some time ago, it seemed povray
3.7 fail to find roots ). Thank you for your support :-D

Post a reply to this message

Am 20.03.2019 um 19:27 schrieb William F Pokorny:

> I can confirm the v38 issue. Also running Ubuntu 18.04 though g++7 vs 
> Warren's 8.
> 
> It seems the v3.8 branch (commit 74b3ebe) needs a space ahead of the 
> macro #end statement to work as follows:
> 
> #macro LoadThing(filePath)
>      #include filePath
>   #end                 // <-- Add extra space before #end and OK
> 
> LoadThing("zoomBangCrash.inc")
> 
> and whenever the include file has an empty line, C++ comment style line, 
> C++ comment line and then a declare line. The following for 
> zoomBangCrash.inc, less the line numbers, is also a v3.8 parse issue if 
> there is no space ahead of the main file macro #end:
> 
> 01
> 02 //
> 03 //
> 04 #declare A=1;

Can't reproduce on Windows; neither with DOS line endings (CR+LF) nor 
with Unix line endings (LF).

I'm running a slightly modified version, but if any of those 
modifications make any difference, I wouldn't know which ones.

Do you have the time and energy to investigate this further?

Post a reply to this message

On 3/21/19 3:18 PM, clipka wrote:
> Am 20.03.2019 um 19:27 schrieb William F Pokorny:
> 
...
> 
> Can't reproduce on Windows; neither with DOS line endings (CR+LF) nor 
> with Unix line endings (LF).
> 
> I'm running a slightly modified version, but if any of those 
> modifications make any difference, I wouldn't know which ones.
> 
> Do you have the time and energy to investigate this further?

Not today with a family visit later, but I'll put it on the list to play 
with the issue more this weekend.

Bill P.

Post a reply to this message

On 3/22/19 6:44 AM, William F Pokorny wrote:
> On 3/21/19 3:18 PM, clipka wrote:
>> Am 20.03.2019 um 19:27 schrieb William F Pokorny:
>>
> ...
>>
>> Can't reproduce on Windows; neither with DOS line endings (CR+LF) nor 
>> with Unix line endings (LF).
>>
>> I'm running a slightly modified version, but if any of those 
>> modifications make any difference, I wouldn't know which ones.
>>
>> Do you have the time and energy to investigate this further?
> 
> Not today with a family visit later, but I'll put it on the list to play 
> with the issue more this weekend.
> 
> 

That was painful to run down, but changing Parser::IsEndOfInvokedMacro() 
in the file source/parser/parser_tokenizer.cpp as in the attached file 
fixes the issue.

Have to admit I'm not sure even now I completely understand what was 
happening (how it could happen perhaps). We seem to have been comparing 
structures in the original - which I don't think c++ does by default?

return (Cond_Stack.back().PMac->endPosition ==
CurrentFilePosition()) && ...

If this true though, why no compile warning or error? In any case, once 
in a while on the structure compare some error got thrown and caught by 
the parser code and no message was set which is why we get the unhelpful 
generic something wrong message. Mostly calls to the compare above 
seemed to work - including ones prior to the failing one for the test 
case - returning false always.

I'm tempted to add a throw on a true test for the re-coded version then 
run a bunch of stuff to see if it true ever really happens. Have we been 
doing something like comparing pointers (and so false) always? But, I've 
already spent a day on this and I don't feel like spending additional 
time.

For the particular test cases you can also hard code that function to 
return false always and things work because that test is always false 
for it.

Lastly, I did try a debug -Og compile thinking maybe we had some 
compiler issue here, but it failed in the same way.

Anyone who better understands what happened / how direct struct compares 
could sort of seem to work, please feel free to jump in and explain.

Bill P.

Post a reply to this message

Attachments:
Download 'isendofinvokedmacro_update.txt' (1 KB)

Am 24.03.2019 um 15:59 schrieb William F Pokorny:

> That was painful to run down, but changing Parser::IsEndOfInvokedMacro() 
> in the file source/parser/parser_tokenizer.cpp as in the attached file 
> fixes the issue.

Well, that's a lead I might investigate. But as you don't seem to be 
confidently understanding what's going on, I'll be going with the 
presumption that this is not a proper fix but just a patch.

> Have to admit I'm not sure even now I completely understand what was 
> happening (how it could happen perhaps). We seem to have been comparing 
> structures in the original - which I don't think c++ does by default?
> 
> return (Cond_Stack.back().PMac->endPosition ==
> CurrentFilePosition()) && ...
> 
> If this true though, why no compile warning or error?

That's easily explained: The data is of type `LexemePosition`, which is 
a struct defining an overloaded comparison operator.

(In C++, structs and classes are the same category of beasts. The only 
semantic difference between `struct` and `class` is the default member 
and base class access: `struct` implies `public` by default, while 
`class` implies `private` by default. According to the language 
standard, it is even perfectly valid to pre-declare a type as `struct` 
and later define it as `class` or vice versa.)

The `LexemePosition` type holds a file offset, as well as a line and 
column number.

Comparison is done by file offset, but the code also features an 
assertion, which tests whether the comparison by line and column yields 
the same result. If the assertion fails, debug builds bomb (i.e. core 
dump or break into a debugger), and non-debug builds throw an exception 
to trigger a parse error.

(Usually such assertion tests are only enabled in debug builds, but in 
the parser they're currently deliberately enabled in all builds, because 
I don't fully trust my current understanding of the parser and the 
refactoring work I have based on that understanding.)

If `Cond_Stack.back().PMac->endPosition == CurrentFilePosition()` throws 
an exception, it means either of two things:

(A) The positions compared are from two different files, which means the 
calling code has failed to check whether it is even in the right file.

(B) The line/column tracking is buggy, and yields different positions in 
different situations.


> In any case, once 
> in a while on the structure compare some error got thrown and caught by 
> the parser code and no message was set which is why we get the unhelpful 
> generic something wrong message.

This is the default error message for failed assertions. Unfortunately 
somewhere along the chain of error message handling the location 
information for the assertion seems to be lost, the original exception 
should include the nformation that it was triggered in `parsertypes.h` 
line 62 (in my version of the file at any rate), maybe even that it was 
in function `LexemePosition::operator==` (depending on compiler). Maybe 
it's worth investigating and fixing that.

Post a reply to this message

Am 25.03.2019 um 23:55 schrieb clipka:

> If `Cond_Stack.back().PMac->endPosition == CurrentFilePosition()` throws 
> an exception, it means either of two things:
> 
> (A) The positions compared are from two different files, which means the 
> calling code has failed to check whether it is even in the right file.
> 
> (B) The line/column tracking is buggy, and yields different positions in 
> different situations.


Well, looking at the code of `IsEndOfInvokedMacro()` again, it 
_strongly_ reeks of scenario (A):

     return (Cond_Stack.back().PMac->endPosition ==
                 CurrentFilePosition()) &&
            (Cond_Stack.back().PMac->source.fileName ==
                 mTokenizer.GetInputStreamName());

should probably be swapped around, like so:

     return (Cond_Stack.back().PMac->source.fileName ==
                 mTokenizer.GetInputStreamName()) &&
            (Cond_Stack.back().PMac->endPosition ==
                 CurrentFilePosition());

so that the `LexemePosition` comparison operator is only invoked if the 
file names match.


I guess you got really lucky in your test (and the OP unlucky in their 
use case), in that you happen to have created the following scenario:

- A macro is invoked.
- That macro includes a different file.
- That included file happens to have a `#` (e.g. `#declare`) at the same 
file offset as the `#end` of the macro.
- However, the `#` happens to be in a different line and/or column.


In such a scenario, encountering the `#`, the parser would first test 
whether this is the `#end` of the macro it has been waiting for, calling 
`IsEndOfInvokedMacro()`.

`IsEndOfInvokedMacro()` in turn would checks whether the current file 
position matches that of the macro's `#end` by calling 
`LexemePostition::operator==`.

`LexemePostition::operator==` in turn would find that the file position 
does match. It would then verify that the match also reflects in line 
and column, and find that they differ, and report this inconsistency via 
an Exception.

`IsEndOfInvokedMacro()` wouldn't even get a chance to test whether it is 
looking at the right file, because its author (duh, I wonder who that 
stupid git might have been...) chose to do the two tests in the wrong order.


Now aside from the stupid git who authored `IsEndOfInvokedMacro()` 
there's someone else to blame for the situation, namely whoever thought 
it was a good idea to keep the stack of include files separate from the 
stack of pending "conditions". If they were using one and the same 
structure, having `INVOKING_MACRO_COND` on top of the stack would imply 
that we're in the file with the `#end`, and the file name test would be 
redundant.

I think it should be possible to merge the include stack into the 
"condition" stack without adverse effects, but I haven't finished work 
on that yet.


I should note that I haven't tested this hypothesis and fix for the 
error yet; I intend to do that as soon as I find the time, but of course 
you're welcome to beat me to it.

Post a reply to this message

On 3/25/19 7:41 PM, clipka wrote:
> Am 25.03.2019 um 23:55 schrieb clipka:
> 
>> If `Cond_Stack.back().PMac->endPosition == CurrentFilePosition()` 
>> throws an exception, it means either of two things:
>>
>> (A) The positions compared are from two different files, which means 
>> the calling code has failed to check whether it is even in the right 
>> file.
>>
>> (B) The line/column tracking is buggy, and yields different positions 
>> in different situations.
> 
> 
> Well, looking at the code of `IsEndOfInvokedMacro()` again, it 
> _strongly_ reeks of scenario (A):
> 
>      return (Cond_Stack.back().PMac->endPosition ==
>                  CurrentFilePosition()) &&
>             (Cond_Stack.back().PMac->source.fileName ==
>                  mTokenizer.GetInputStreamName());
> 
> should probably be swapped around, like so:
> 
>      return (Cond_Stack.back().PMac->source.fileName ==
>                  mTokenizer.GetInputStreamName()) &&
>             (Cond_Stack.back().PMac->endPosition ==
>                  CurrentFilePosition());
> 
> so that the `LexemePosition` comparison operator is only invoked if the 
> file names match.
> 
> 
> I guess you got really lucky in your test (and the OP unlucky in their 
> use case), in that you happen to have created the following scenario:
> 
> - A macro is invoked.
> - That macro includes a different file.
> - That included file happens to have a `#` (e.g. `#declare`) at the same 
> file offset as the `#end` of the macro.
> - However, the `#` happens to be in a different line and/or column.
> 
> 
> In such a scenario, encountering the `#`, the parser would first test 
> whether this is the `#end` of the macro it has been waiting for, calling 
> `IsEndOfInvokedMacro()`.
> 
> `IsEndOfInvokedMacro()` in turn would checks whether the current file 
> position matches that of the macro's `#end` by calling 
> `LexemePostition::operator==`.
> 
> `LexemePostition::operator==` in turn would find that the file position 
> does match. It would then verify that the match also reflects in line 
> and column, and find that they differ, and report this inconsistency via 
> an Exception.
> 
> `IsEndOfInvokedMacro()` wouldn't even get a chance to test whether it is 
> looking at the right file, because its author (duh, I wonder who that 
> stupid git might have been...) chose to do the two tests in the wrong 
> order.
> 
> 
> Now aside from the stupid git who authored `IsEndOfInvokedMacro()` 
> there's someone else to blame for the situation, namely whoever thought 
> it was a good idea to keep the stack of include files separate from the 
> stack of pending "conditions". If they were using one and the same 
> structure, having `INVOKING_MACRO_COND` on top of the stack would imply 
> that we're in the file with the `#end`, and the file name test would be 
> redundant.
> 
> I think it should be possible to merge the include stack into the 
> "condition" stack without adverse effects, but I haven't finished work 
> on that yet.
> 
> 
> I should note that I haven't tested this hypothesis and fix for the 
> error yet; I intend to do that as soon as I find the time, but of course 
> you're welcome to beat me to it.

Thanks for the detailed explanations and, though I'm busy with RL 
starting in a little while today, I think you've figured it out.

I know from looking at the values the position was matching and the line 
and column were not. Further, when I look at my test cases (all some 
pretty close variation of Warren's original) the failing ones all have 
the # of the declare in just the right spot as you suggested. I'd not 
recognized this was so.

My patch, then, just avoided the assertion in the 
LexemePosition::operator== I suppose. I also didn't know the parser 
assertions were today on always and I wasn't thinking an assert could 
cause the exception.

Quick additional question. Is it the --enable-debug configure flag that 
- normally - turns on the pov asserts? I've never verified this 
assumption of mine by creating a pov assert in the code I know should 
fail.

Bill P.

Post a reply to this message

Am 26.03.2019 um 11:46 schrieb William F Pokorny:

> Quick additional question. Is it the --enable-debug configure flag that 
> - normally - turns on the pov asserts? I've never verified this 
> assumption of mine by creating a pov assert in the code I know should fail.

To be honest, I haven't a clue. I'm not sure what the `--enable-debug` 
flag does exactly (it's some standard mechanism in the Automake tools), 
nor whether and how assertions are enabled or disabled in Unix builds.

I'm pretty sure that `--enable-debug` does enable the creation of debug 
information in the binaries, so that a debugging tool can correlate 
binary code addresses with original source code lines, so that you can 
e.g. set breakpoints and step through the code line by line with a 
debugger. I also wouldn't be surprised if it turns off optimizations by 
default, because those tend to re-order instructions and make a 
line-by-line correlation impossible. Whether it does anything beyond 
that I do not know. It might dictate whether `NDEBUG` is defined or not, 
which would conceivably disable/enable assertions.

Post a reply to this message