Please give this version a thorough shakedown, with focus on parsing:

https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999

Besides testing functionality, please also look at parsing speed.


Known issues:

- `#read` currently disabled
- macro caching currently disabled
- system-specific character encoding in strings currently not supported
(but utf8 should work)
- signature BOM in utf8-encoded files currently not supported
- probably one or two things I'm forgetting to mention

- Backward compatibility with scenes that use single backslashes in
literal filenames has been sacrificed, and will most likely not be restored.


The parser overhaul has been extended to cover not just the "scanner"
stage, but also "raw tokenization", whereby literals are already
evaluated (e.g. digit sequences converted to numbers, and escape
sequences in strings resolved) and reserved words identified. Pretty
much the only tokenization-related task remaining in the monolithic bulk
of the parser is identifying variables, which requires knowledge and
understanding of context.

In passing, a few internal limitations of the parser have also been
lifted; for instance, there is no longer any fixed limit to the nesting
depth of include files, or the nesting depth of parentheses, braces etc.

Post a reply to this message

On 22-5-2018 2:28, clipka wrote:
> Please give this version a thorough shakedown, with focus on parsing:

A test of a scene comprising a height_field, media clouds, and 4000 
cubes distributed on the hf (a scene I am working on at this moment). 
Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71; 
(3) version 3.71.

See attachment for parser details.

Is this what you want?

-- 
Thomas

Post a reply to this message

Attachments:
Download 'testing pov-ray.txt' (4 KB)

Thomas de Groot <tho### [at] degrootorg> wrote:
> On 22-5-2018 2:28, clipka wrote:
> > Please give this version a thorough shakedown, with focus on parsing:
>
> A test of a scene comprising a height_field, media clouds, and 4000
> cubes distributed on the hf (a scene I am working on at this moment).
> Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71;
> (3) version 3.71.
>
> See attachment for parser details.
>
> Is this what you want?

I'm not really interested in individual scenes, unless they demonstrate a
problem with the new version. More a general "wow that's fast" or "uh, that's
slow" feedback.

Also, when testing parser performance, remember parsing a "cold" file tends to
take longer than a "hot" one.

Post a reply to this message

On 22-5-2018 12:46, clipka wrote:
> Thomas de Groot <tho### [at] degrootorg> wrote:
>> On 22-5-2018 2:28, clipka wrote:
>>> Please give this version a thorough shakedown, with focus on parsing:
>>
>> A test of a scene comprising a height_field, media clouds, and 4000
>> cubes distributed on the hf (a scene I am working on at this moment).
>> Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71;
>> (3) version 3.71.
>>
>> See attachment for parser details.
>>
>> Is this what you want?
> 
> I'm not really interested in individual scenes, unless they demonstrate a
> problem with the new version. More a general "wow that's fast" or "uh, that's
> slow" feedback.

Ok. So it this particular case, the parser was slower (29s. vs 4sec).

> 
> Also, when testing parser performance, remember parsing a "cold" file tends to
> take longer than a "hot" one.
> 

Can you explain hot vs cold?

-- 
Thomas

Post a reply to this message

> > Also, when testing parser performance, remember parsing a "cold" file tends to
> > take longer than a "hot" one.
> >
>
> Can you explain hot vs cold?

By "hot" I mean recently read (by this or any other application), increasing the
probability that it is still cached somewhere and doesn't have to be read
completely new from disk.

Post a reply to this message

On 05/21/2018 08:28 PM, clipka wrote:
> Please give this version a thorough shakedown, with focus on parsing:
> 
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999
> 
> Besides testing functionality, please also look at parsing speed.
> 
...
> 

I've not got a lot of POV-Ray time today, but ran a few random quick 
parse scenes - all OK. Then three sets using the Harmonograph.pov
code posted to the newsgroups early last year - loops of the type we 
want sped up. Comparing master to your tagged release. All testing done 
on a ramdisk though doesn't matter much in the cases I ran. Ubuntu 16.04 
g++ 5.4.0.

New parser faster :

28.80 -> 26.60 ---> -7.64%
28.80 -> 26.34 ---> -8.54%
29.24 -> 26.36 ---> -9.85%

Bill P.

Post a reply to this message

Am 22.05.2018 um 12:57 schrieb Thomas de Groot:

> Ok. So it this particular case, the parser was slower (29s. vs 4sec).

Hm... that's actually quite the difference.

Does the scene in question perhaps rely heavily on macros defined in
another file? Then the disabled macro caching might be a fitting
explanation.


Also, care to share the scene?

Post a reply to this message

Am 22.05.2018 um 15:35 schrieb William F Pokorny:

> I've not got a lot of POV-Ray time today, but ran a few random quick
> parse scenes - all OK. Then three sets using the Harmonograph.pov
> code posted to the newsgroups early last year - loops of the type we
> want sped up. Comparing master to your tagged release. All testing done
> on a ramdisk though doesn't matter much in the cases I ran. Ubuntu 16.04
> g++ 5.4.0.
> 
> New parser faster :
> 
> 28.80 -> 26.60 ---> -7.64%
> 28.80 -> 26.34 ---> -8.54%
> 29.24 -> 26.36 ---> -9.85%

Now that's what I call better news than those from the country of cheese ;)

And I haven't even really started working on the mechanism by which I
intend to speed up those loops.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Please give this version a thorough shakedown, with focus on parsing:
>
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999
>
> Besides testing functionality, please also look at parsing speed.
>
>
i tried this versus uberpov, win7 64bits.  uberpov first at 7m44s (parse only),
with x.tokenizer immediately after (for whatever that means cold/hot) at 7m42s.
essentially identical.

Post a reply to this message

On 22-5-2018 16:31, clipka wrote:
> Am 22.05.2018 um 12:57 schrieb Thomas de Groot:
> 
>> Ok. So it this particular case, the parser was slower (29s. vs 4sec).
> 
> Hm... that's actually quite the difference.
> 
> Does the scene in question perhaps rely heavily on macros defined in
> another file? Then the disabled macro caching might be a fitting
> explanation.
> 
> 
> Also, care to share the scene?
> 

 From the Land of Cheese:
Where the longer parse time comes from is in the while loop distributing 
the cubes. This loop resides in a separate inc file. and looks like this:

//==========================================================
#local R1 = seed(1929);
#local R2 = seed(9732);
#local R3 = seed(6493);
#local N1 = 4000;
#local I  = 0;

#while (I <= N1)
   #local Start = VRand_In_Obj(RocksBox, R1);
   #local Norm = <0, 0, 0>;
   #local RockPos = trace(Landscape, Start, -y, Norm);
   #declare Visible = IsObjectVisible(RockPos)
   #if (Visible)
     #if (vdot(Norm, y)>0.8 & RockPos.y<50.0*(0.8+rand(R1)*0.4) & 
RockF(RockPos.x,RockPos.y,RockPos.z)<0.5+rand(R1)*0.1 & RockPos.y>0 )
       object {Rock
         //scale (RockPos.y+(1/RockPos.y))*RRand(0.9, 1.1, R2)
         scale RockPos.y*RRand(0.9, 1.1, R2)
         rotate RRand(-180, 180, R3)
         translate RockPos
       }
       #if (mod(I, N1/10) = 0)
         #debug concat(" Rocks: ", str(I,5,0), "\n")
       #end
       #local I = I + 1;
     #end  //slope & altitude
   #end  //Visible
#end
//==========================================================

Otherwise, parse time is about equal between versions when the loop is 
disabled.

-- 
Thomas

Post a reply to this message