On 1/14/19 9:07 AM, William F Pokorny wrote:
> On 1/13/19 5:07 PM, clipka wrote:
...
> 

Just noticed another interesting thing in the tests that did run. The 
attached file has not parser cleanly since v3.6.1. Generating the 
following error with v3.7:

File 'Sig07_works_v361_earlier.pov' line 1: Parse Error: Unexpected 
additional
  '.' in floating-point number
Fatal error in parser: Cannot parse input.
Render failed


It parses and runs cleanly again with the updated parser. Intentional?

Bill P.

Post a reply to this message

Attachments:
Download 'sig07_works_v361_earlier.pov.txt' (1 KB)

Am 14.01.2019 um 16:18 schrieb William F Pokorny:

> Just noticed another interesting thing in the tests that did run. The 
> attached file has not parser cleanly since v3.6.1. Generating the 
> following error with v3.7:

I presume you mean to say that it worked in v3.6.1, but not in v3.7, 
correct?

> File 'Sig07_works_v361_earlier.pov' line 1: Parse Error: Unexpected 
> additional
>   '.' in floating-point number
> Fatal error in parser: Cannot parse input.
> Render failed

Those are some mean pieces of scene code; no surprise they catch the 
parser with its pants down ;)

> It parses and runs cleanly again with the updated parser. Intentional?

No, just a lucky punch. Turns out sometimes throwing away and rewriting 
old code fixes things you didn't even know were broken.

Post a reply to this message

Am 14.01.2019 um 15:07 schrieb William F Pokorny:

> File 'Sig00.pov' line 2: Parse Warning: Float value promoted to full 
> color vector where both filter and transmit >0.0.
> File 'Sig00.pov' line 1: Parse Error: End of file reached but #end 
> expected.
> Fatal error in parser: Cannot parse input.
> Render failed

Traced this down to a bug in the scanner that kicks in whenever two 
consecutive returns from a macro or include file jump back to the same 
binary offset in (formally) different input streams

One such scenario is the return from a macro that has called itself 
recursively and hasn't called other macros since.

Post a reply to this message

On 1/14/19 10:54 AM, clipka wrote:
> Am 14.01.2019 um 16:18 schrieb William F Pokorny:
> 
>> Just noticed another interesting thing in the tests that did run. The 
>> attached file has not parser cleanly since v3.6.1. Generating the 
>> following error with v3.7:
> 
> I presume you mean to say that it worked in v3.6.1, but not in v3.7, 
> correct?

Yes, I did! You are getting good at understand my gibberish. :-)

> 
...
> 
>> It parses and runs cleanly again with the updated parser. Intentional?
> 
> No, just a lucky punch. Turns out sometimes throwing away and rewriting 
> old code fixes things you didn't even know were broken.

Suppose my question was too whether we should consider the 3.7/3.8 
behavior broken - you seem to be saying it was.

Folks can again cause parsing to separate values by using extra decimal 
points in numeric strings. Has me wondering what might happen with typos 
like 1..2 (not tried it). Or where we specify lists of values separated 
by commas or not. Can the latter get scrambled in ways that are confusing?

I'll go try some things I guess to better understand what can again 
happen with this reversion in parser behavior.

Bill P.

Post a reply to this message

On 1/14/19 11:39 AM, clipka wrote:
> Am 14.01.2019 um 15:07 schrieb William F Pokorny:
> 
>> File 'Sig00.pov' line 2: Parse Warning: Float value promoted to full 
>> color vector where both filter and transmit >0.0.
>> File 'Sig00.pov' line 1: Parse Error: End of file reached but #end 
>> expected.
>> Fatal error in parser: Cannot parse input.
>> Render failed
> 
> Traced this down to a bug in the scanner that kicks in whenever two 
> consecutive returns from a macro or include file jump back to the same 
> binary offset in (formally) different input streams
> 
> One such scenario is the return from a macro that has called itself 
> recursively and hasn't called other macros since.

Cool. Glad you found it.

Bill P.

Post a reply to this message

On 1/13/19 5:07 PM, clipka wrote:
> Am 13.01.2019 um 23:04 schrieb clipka:
>> Another known issue:
...
> 
> Users building their own binaries (most notably Unix users) may also use 
> `source/core/shape/truetype.cpp` from the latest master, which should 
> fix the issue:
> 
>
https://raw.githubusercontent.com/POV-Ray/povray/master/source/core/shape/truetype.cpp

> 

Stuff still running. I've turned up several more of the macro/include 
return type.

One of our standard shipped scene files fails due a form feed character 
'0c' very near the end the end of the file. That scene is:

.../distribution/scenes/objects/superel2.pov

and here I think we just need to update that scene file.

Bill P.

Post a reply to this message

Am 14.01.2019 um 18:15 schrieb William F Pokorny:

> Suppose my question was too whether we should consider the 3.7/3.8 
> behavior broken - you seem to be saying it was.

Let's put it this way: Going by the reference docs on numeric 
expressions - specifically FLOAT_LITERAL - the v3.7 behaviour would have 
to be considered broken, unless interpreting the syntax in a 
self-inconsistent manner.

The key question is under what circumstances a sequence of characters 
that matches the production rules of FLOAT_LITERAL, followed by some 
other character that does not fit in, constitutes a broken 
FLOAT_LITERAL, as opposed to a valid FLOAT_LITERAL token immediately 
followed by the next token.

The v3.6 behaviour can easily be interpreted as consistently treating 
such cases as valid, but erroneously implementing MANTISSA as:

     MANTISSA:
       DIGIT... [DECIMAL_POINT | [DIGIT2...] |
       DECIMAL_POINT DIGIT2...
     DIGIT2:
       DIGIT | DECIMAL_POINT

The v3.7 behaviour must be interpreted as treating such cases as broken 
if the non-matching character is a dot, but valid if it is any other 
character. That's a pretty arbitrary distinction.

The (new) v3.8 behaviour consistently treats all such cases as valid 
again, while implementing MANTISSA as documented.


> Folks can again cause parsing to separate values by using extra decimal 
> points in numeric strings. Has me wondering what might happen with typos 
> like 1..2 (not tried it).

That would have to be `1.` followed by `.2`.

> Or where we specify lists of values separated 
> by commas or not. Can the latter get scrambled in ways that are confusing?

Dots don't constitute valid infix or postfix operators in numeric 
expressions, so there's no ambiguity there.

Post a reply to this message

On 1/14/19 1:41 PM, clipka wrote:
> Am 14.01.2019 um 18:15 schrieb William F Pokorny:
> 
>> Suppose my question was too whether we should consider the 3.7/3.8 
>> behavior broken - you seem to be saying it was.
> 
> Let's put it this way: Going by the reference docs on numeric 
> expressions - specifically FLOAT_LITERAL - the v3.7 behaviour would have 
> to be considered broken, unless interpreting the syntax in a 
> self-inconsistent manner.
> 
> The key question is under what circumstances a sequence of characters 
> that matches the production rules of FLOAT_LITERAL, followed by some 
> other character that does not fit in, constitutes a broken 
> FLOAT_LITERAL, as opposed to a valid FLOAT_LITERAL token immediately 
> followed by the next token.
> 
> The v3.6 behaviour can easily be interpreted as consistently treating 
> such cases as valid, but erroneously implementing MANTISSA as:
> 
>      MANTISSA:
>        DIGIT... [DECIMAL_POINT | [DIGIT2...] |
>        DECIMAL_POINT DIGIT2...
>      DIGIT2:
>        DIGIT | DECIMAL_POINT
> 
> The v3.7 behaviour must be interpreted as treating such cases as broken 
> if the non-matching character is a dot, but valid if it is any other 
> character. That's a pretty arbitrary distinction.
> 
> The (new) v3.8 behaviour consistently treats all such cases as valid 
> again, while implementing MANTISSA as documented.
> 
> 
>> Folks can again cause parsing to separate values by using extra 
>> decimal points in numeric strings. Has me wondering what might happen 
>> with typos like 1..2 (not tried it).
> 
> That would have to be `1.` followed by `.2`.
> 
>> Or where we specify lists of values separated by commas or not. Can 
>> the latter get scrambled in ways that are confusing?
> 
> Dots don't constitute valid infix or postfix operators in numeric 
> expressions, so there's no ambiguity there.

Thanks for the detail. I've been playing with variations for the last 
hour or so and created some new parser test cases in my collection to 
track this change. While 1..2 gets treated as two numbers .1..2 and the 
like do not - so even the exposure to period over comma typos is small.

Not picked up anything new and have switched over to your latest code - 
it fixed all the recursive return fails I had. Should finish up tomorrow 
sometime and will post again only if something new turns up.

For the record I'm only checking the parser with my tests. I'm not doing 
image to image compares. Image compares will happen over time after I 
rebase my branches onto the current master head and continue other 
efforts.

Bill P.

Post a reply to this message

Am 14.01.2019 um 21:29 schrieb William F Pokorny:

> For the record I'm only checking the parser with my tests. I'm not doing 
> image to image compares. Image compares will happen over time after I 
> rebase my branches onto the current master head and continue other efforts.

I think that's perfectly fine; I don't expect any parser issues to be 
subtle enough to make it to the rendering stage - with the exception of 
string and text handling, specifically non-ASCII stuff.

Post a reply to this message

Am 14.01.2019 um 18:23 schrieb William F Pokorny:

> One of our standard shipped scene files fails due a form feed character 
> '0c' very near the end the end of the file. That scene is:
> 
> ..../distribution/scenes/objects/superel2.pov
> 
> and here I think we just need to update that scene file.

Page breaks used to be common in ASCII files a while ago; I feel somehow 
inclined to just allow them as whitespace.

The only thing that's holding me back is that I'm not exactly sure what 
effect they should have on column and line counting.

Post a reply to this message