Please give this version a thorough shakedown, with focus on parsing:

https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999

Besides testing functionality, please also look at parsing speed.


Known issues:

- `#read` currently disabled
- macro caching currently disabled
- system-specific character encoding in strings currently not supported
(but utf8 should work)
- signature BOM in utf8-encoded files currently not supported
- probably one or two things I'm forgetting to mention

- Backward compatibility with scenes that use single backslashes in
literal filenames has been sacrificed, and will most likely not be restored.


The parser overhaul has been extended to cover not just the "scanner"
stage, but also "raw tokenization", whereby literals are already
evaluated (e.g. digit sequences converted to numbers, and escape
sequences in strings resolved) and reserved words identified. Pretty
much the only tokenization-related task remaining in the monolithic bulk
of the parser is identifying variables, which requires knowledge and
understanding of context.

In passing, a few internal limitations of the parser have also been
lifted; for instance, there is no longer any fixed limit to the nesting
depth of include files, or the nesting depth of parentheses, braces etc.

Post a reply to this message

On 22-5-2018 2:28, clipka wrote:
> Please give this version a thorough shakedown, with focus on parsing:

A test of a scene comprising a height_field, media clouds, and 4000 
cubes distributed on the hf (a scene I am working on at this moment). 
Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71; 
(3) version 3.71.

See attachment for parser details.

Is this what you want?

-- 
Thomas

Post a reply to this message

Attachments:
Download 'testing pov-ray.txt' (4 KB)

Thomas de Groot <tho### [at] degrootorg> wrote:
> On 22-5-2018 2:28, clipka wrote:
> > Please give this version a thorough shakedown, with focus on parsing:
>
> A test of a scene comprising a height_field, media clouds, and 4000
> cubes distributed on the hf (a scene I am working on at this moment).
> Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71;
> (3) version 3.71.
>
> See attachment for parser details.
>
> Is this what you want?

I'm not really interested in individual scenes, unless they demonstrate a
problem with the new version. More a general "wow that's fast" or "uh, that's
slow" feedback.

Also, when testing parser performance, remember parsing a "cold" file tends to
take longer than a "hot" one.

Post a reply to this message

On 22-5-2018 12:46, clipka wrote:
> Thomas de Groot <tho### [at] degrootorg> wrote:
>> On 22-5-2018 2:28, clipka wrote:
>>> Please give this version a thorough shakedown, with focus on parsing:
>>
>> A test of a scene comprising a height_field, media clouds, and 4000
>> cubes distributed on the hf (a scene I am working on at this moment).
>> Tested with (1) the experimental 3.8 version; (2) UberPOV patch 3.71;
>> (3) version 3.71.
>>
>> See attachment for parser details.
>>
>> Is this what you want?
> 
> I'm not really interested in individual scenes, unless they demonstrate a
> problem with the new version. More a general "wow that's fast" or "uh, that's
> slow" feedback.

Ok. So it this particular case, the parser was slower (29s. vs 4sec).

> 
> Also, when testing parser performance, remember parsing a "cold" file tends to
> take longer than a "hot" one.
> 

Can you explain hot vs cold?

-- 
Thomas

Post a reply to this message

> > Also, when testing parser performance, remember parsing a "cold" file tends to
> > take longer than a "hot" one.
> >
>
> Can you explain hot vs cold?

By "hot" I mean recently read (by this or any other application), increasing the
probability that it is still cached somewhere and doesn't have to be read
completely new from disk.

Post a reply to this message

On 05/21/2018 08:28 PM, clipka wrote:
> Please give this version a thorough shakedown, with focus on parsing:
> 
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999
> 
> Besides testing functionality, please also look at parsing speed.
> 
...
> 

I've not got a lot of POV-Ray time today, but ran a few random quick 
parse scenes - all OK. Then three sets using the Harmonograph.pov
code posted to the newsgroups early last year - loops of the type we 
want sped up. Comparing master to your tagged release. All testing done 
on a ramdisk though doesn't matter much in the cases I ran. Ubuntu 16.04 
g++ 5.4.0.

New parser faster :

28.80 -> 26.60 ---> -7.64%
28.80 -> 26.34 ---> -8.54%
29.24 -> 26.36 ---> -9.85%

Bill P.

Post a reply to this message

Am 22.05.2018 um 12:57 schrieb Thomas de Groot:

> Ok. So it this particular case, the parser was slower (29s. vs 4sec).

Hm... that's actually quite the difference.

Does the scene in question perhaps rely heavily on macros defined in
another file? Then the disabled macro caching might be a fitting
explanation.


Also, care to share the scene?

Post a reply to this message

Am 22.05.2018 um 15:35 schrieb William F Pokorny:

> I've not got a lot of POV-Ray time today, but ran a few random quick
> parse scenes - all OK. Then three sets using the Harmonograph.pov
> code posted to the newsgroups early last year - loops of the type we
> want sped up. Comparing master to your tagged release. All testing done
> on a ramdisk though doesn't matter much in the cases I ran. Ubuntu 16.04
> g++ 5.4.0.
> 
> New parser faster :
> 
> 28.80 -> 26.60 ---> -7.64%
> 28.80 -> 26.34 ---> -8.54%
> 29.24 -> 26.36 ---> -9.85%

Now that's what I call better news than those from the country of cheese ;)

And I haven't even really started working on the mechanism by which I
intend to speed up those loops.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Please give this version a thorough shakedown, with focus on parsing:
>
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999
>
> Besides testing functionality, please also look at parsing speed.
>
>
i tried this versus uberpov, win7 64bits.  uberpov first at 7m44s (parse only),
with x.tokenizer immediately after (for whatever that means cold/hot) at 7m42s.
essentially identical.

Post a reply to this message

On 22-5-2018 16:31, clipka wrote:
> Am 22.05.2018 um 12:57 schrieb Thomas de Groot:
> 
>> Ok. So it this particular case, the parser was slower (29s. vs 4sec).
> 
> Hm... that's actually quite the difference.
> 
> Does the scene in question perhaps rely heavily on macros defined in
> another file? Then the disabled macro caching might be a fitting
> explanation.
> 
> 
> Also, care to share the scene?
> 

 From the Land of Cheese:
Where the longer parse time comes from is in the while loop distributing 
the cubes. This loop resides in a separate inc file. and looks like this:

//==========================================================
#local R1 = seed(1929);
#local R2 = seed(9732);
#local R3 = seed(6493);
#local N1 = 4000;
#local I  = 0;

#while (I <= N1)
   #local Start = VRand_In_Obj(RocksBox, R1);
   #local Norm = <0, 0, 0>;
   #local RockPos = trace(Landscape, Start, -y, Norm);
   #declare Visible = IsObjectVisible(RockPos)
   #if (Visible)
     #if (vdot(Norm, y)>0.8 & RockPos.y<50.0*(0.8+rand(R1)*0.4) & 
RockF(RockPos.x,RockPos.y,RockPos.z)<0.5+rand(R1)*0.1 & RockPos.y>0 )
       object {Rock
         //scale (RockPos.y+(1/RockPos.y))*RRand(0.9, 1.1, R2)
         scale RockPos.y*RRand(0.9, 1.1, R2)
         rotate RRand(-180, 180, R3)
         translate RockPos
       }
       #if (mod(I, N1/10) = 0)
         #debug concat(" Rocks: ", str(I,5,0), "\n")
       #end
       #local I = I + 1;
     #end  //slope & altitude
   #end  //Visible
#end
//==========================================================

Otherwise, parse time is about equal between versions when the loop is 
disabled.

-- 
Thomas

Post a reply to this message

On 22/05/2018 01:28, clipka wrote:
> - Backward compatibility with scenes that use single backslashes in
> literal filenames has been sacrificed, and will most likely not be restored.

This is going to cause me problems using SDL generated by modellers.

Test:

v 3.8.0-alpha.9475849+av541.msvc14 is faster than
v 3.8.0-x. tokenizer.9999+av5 60.msvc

On the second run of the file. The parse time was a second or two slower 
than the first time. Except when using Ver 3.7 when it was the same.

Statistic Files attached



-- 

Regards
     Stephen

Post a reply to this message

Attachments:
Download 'hot_v 3.8.0-alpha.9475849+av541.msvc14.txt' (3 KB) Download 'hot_v 3.8.0-x. tokenizer.9999+av5 60.msvc.txt' (3 KB) Download 'hot_version 3.7.0.msvc10.win64_.txt' (3 KB) Download 'v 3.8.0-alpha.9475849+av541.msvc14.txt' (3 KB) Download 'v 3.8.0-x. tokenizer.9999+av5 60.msvc.txt' (3 KB) Download 'version 3.7.0.msvc10.win64_.txt' (3 KB)

Am 23.05.2018 um 08:54 schrieb Thomas de Groot:

> Where the longer parse time comes from is in the while loop distributing
> the cubes. This loop resides in a separate inc file. and looks like this:
> 
> //==========================================================
> #local R1 = seed(1929);
> #local R2 = seed(9732);
> #local R3 = seed(6493);
> #local N1 = 4000;
> #local I  = 0;
> 
> #while (I <= N1)
>   #local Start = VRand_In_Obj(RocksBox, R1);
                   ^^^^^^^^^^^^
Unless you copied that to your file with the loop, that's a macro in a
different file...

>   #local Norm = <0, 0, 0>;
>   #local RockPos = trace(Landscape, Start, -y, Norm);
>   #declare Visible = IsObjectVisible(RockPos)
>   #if (Visible)
>     #if (vdot(Norm, y)>0.8 & RockPos.y<50.0*(0.8+rand(R1)*0.4) &
> RockF(RockPos.x,RockPos.y,RockPos.z)<0.5+rand(R1)*0.1 & RockPos.y>0 )
>       object {Rock
>         //scale (RockPos.y+(1/RockPos.y))*RRand(0.9, 1.1, R2)
                                            ^^^^^
>         scale RockPos.y*RRand(0.9, 1.1, R2)
                          ^^^^^
>         rotate RRand(-180, 180, R3)
                 ^^^^^
... as is this.

>         translate RockPos
>       }
>       #if (mod(I, N1/10) = 0)
>         #debug concat(" Rocks: ", str(I,5,0), "\n")
>       #end
>       #local I = I + 1;
>     #end  //slope & altitude
>   #end  //Visible
> #end

So yes, that pretty much explains the poor performance with the
overhauled parser -- or, more to the point, the better performance with
v3.8.0-alpha: The latter features cached macros, while the overhauled
parser currently has those disabled.

I would expect v3.7.0 to show roughly similar performance as the
overhauled parser.

Post a reply to this message

Am 23.05.2018 um 10:22 schrieb Stephen:
> On 22/05/2018 01:28, clipka wrote:
>> - Backward compatibility with scenes that use single backslashes in
>> literal filenames has been sacrificed, and will most likely not be
>> restored.
> 
> This is going to cause me problems using SDL generated by modellers.

Care to do some name calling?

Techically, those modellers would have to be considered broken: To the
best of my knowledge, single backslashes as filename separators have
never been an officially supported feature.

That said, if the modeller in question should happen to be of reasonable
significance despite no longer being actively maintained, I may go back
to the drawing board after all.

Post a reply to this message

Am 23.05.2018 um 10:22 schrieb Stephen:

> Test:
> 
> v 3.8.0-alpha.9475849+av541.msvc14 is faster than
> v 3.8.0-x. tokenizer.9999+av5 60.msvc
> 
> On the second run of the file. The parse time was a second or two slower
> than the first time. Except when using Ver 3.7 when it was the same.

I take that as good news, presuming the scene in question does call
macros across files. Such a scenario would explain the superior
performance of v3.8.0-alpha over v3.7.0 as the result of macro caching,
as well as the inferior performance of v3.8.0-x.tokenizer vs.
v3.8.0-alpha as the result of disabled macro caching.

What really excites me is the superior performance of v3.8.0-x.tokenizer
over v3.7.0. It means that with macro caching re-enabled it should not
only break even with, but actually surpass, v3.8.0-alpha performance.

Mind you, I haven't even started with the actual speed improvements that
I am aiming for. All I've been doing so far was adding structure (and
thus potential overhead) to the parser while trying to minimize added
overhead. Apparently I've actually reduced some existing overhead along
the way.

Post a reply to this message

Am 22.05.2018 um 02:28 schrieb clipka:
> Please give this version a thorough shakedown, with focus on parsing:
> 
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999
> 
> Besides testing functionality, please also look at parsing speed.

Thanks to all who have tested for parsing speed so far. I'm quite
confident now that v3.8.0-x.tokenizer is on the right track in that
respect, so my main worry now is whether the SDL functionality is all
there (except of course the portions I have deliberately disabled for
now), without any bugs having crept in.

The fact that none of you has reported any bugs yet makes me a bit
uneasy... the new code can't be /that/ good ;)

(Also, I intend to create a collection of test cases for future
regression testing of the parser, and was hoping for lots of bug reports
for inspiration. Looks like that plan is pretty much foiled ;))

Post a reply to this message

On 23/05/2018 13:26, clipka wrote:
> Am 23.05.2018 um 10:22 schrieb Stephen:
>> On 22/05/2018 01:28, clipka wrote:
>>> - Backward compatibility with scenes that use single backslashes in
>>> literal filenames has been sacrificed, and will most likely not be
>>> restored.
>>
>> This is going to cause me problems using SDL generated by modellers.
> 
> Care to do some name calling?
> 

Pal / Best friend. ;-)

> Technically, those modellers would have to be considered broken: 

That seems fair enough.

> To the
> best of my knowledge, single backslashes as filename separators have
> never been an officially supported feature.
> 
By PovRay for Windows and/or for DOS?


> That said, if the modeller in question should happen to be of reasonable
> significance despite no longer being actively maintained, I may go back
> to the drawing board after all.
> 

Significance is relevant. I have no idea how many people use B3D. And I 
can't remember if Blender outputs the directory separator as a "\" or 
"/". (I have stopped using it until some of the bug reports I have 
submitted, have been fixed.)

It is possible to get B3D to export bespoke code. So we could set a flag 
telling the parser that the directory separator was "\".
e.g. windoze_separator = true
If that would help cut down the complexity.

Which reminds me. I use that facility to overwrite the version number 
B3D exports by declaring it after the first one. Will that workaround 
still work?

Best friend. :-)




-- 

Regards
     Stephen

Post a reply to this message

On 23/05/2018 13:55, clipka wrote:
> Am 23.05.2018 um 10:22 schrieb Stephen:
> 
>> Test:
>>
>> v 3.8.0-alpha.9475849+av541.msvc14 is faster than
>> v 3.8.0-x. tokenizer.9999+av5 60.msvc
>>
>> On the second run of the file. The parse time was a second or two slower
>> than the first time. Except when using Ver 3.7 when it was the same.
> 
> I take that as good news, presuming the scene in question does call
> macros across files. Such a scenario would explain the superior
> performance of v3.8.0-alpha over v3.7.0 as the result of macro caching,
> as well as the inferior performance of v3.8.0-x.tokenizer vs.
> v3.8.0-alpha as the result of disabled macro caching.
> 

Sorry it does not use macros. It did but I de-constructed the grass 
macro it was using as it was declaring an array of meshes from within a 
while loop. [Watch the memory usage rise and the HDD thrash.]


I have already had jr complain that I sent him the code in one file and 
not split into include files.
So, what is wrong with 54317 lines in one file, anyway? ;-)


> What really excites me is the superior performance of v3.8.0-x.tokenizer
> over v3.7.0. It means that with macro caching re-enabled it should not
> only break even with, but actually surpass, v3.8.0-alpha performance.
> 
> Mind you, I haven't even started with the actual speed improvements that
> I am aiming for. All I've been doing so far was adding structure (and
> thus potential overhead) to the parser while trying to minimize added
> overhead. Apparently I've actually reduced some existing overhead along
> the way.
> 


-- 

Regards
     Stephen

Post a reply to this message

Am 23.05.2018 um 16:18 schrieb Stephen:

>>> This is going to cause me problems using SDL generated by modellers.
>>
>> Care to do some name calling?
>>
> 
> Pal / Best friend. ;-)

Not exactly what I meant - I was rather trying to coax you into posting
the name of the modeller, thereby calling it broken ;)


>> Technically, those modellers would have to be considered broken: 
> 
> That seems fair enough.
> 
>> To the
>> best of my knowledge, single backslashes as filename separators have
>> never been an officially supported feature.
>>
> By PovRay for Windows and/or for DOS?

Technically, that statement was true for all of them: I know precious
little about POV-Ray for DOS, so to the best of my knowledge... ;)


But I have to revert that statement now, having dug up an older
documentation:

In POV-Ray 3.0, according to those docs, backslashes in string literals
was only supposed to take on a special role when followed by a
double-quote, in which case the backslash would be discarded while the
double-quote would be interpreted as a character rather than the end of
the string. Interpretation of other escape sequences was considered not
a feature of string literals, but rather of user message stream output
(`#debug` and its kin).


> Significance is relevant. I have no idea how many people use B3D. And I
> can't remember if Blender outputs the directory separator as a "\" or
> "/". (I have stopped using it until some of the bug reports I have
> submitted, have been fixed.)

Yeah, I guess B3D qualifies as relevant indeed.


> It is possible to get B3D to export bespoke code. So we could set a flag
> telling the parser that the directory separator was "\".
> e.g. windoze_separator = true
> If that would help cut down the complexity.

Not sure I understand what you mean. Would you change B3D, or would you
change POV-Ray?

When it comes to ideal format in POV-Ray scenes, it is highly
recommended to always use "/" as path separator. Works on all platforms,
'Doze incldued.


> Which reminds me. I use that facility to overwrite the version number
> B3D exports by declaring it after the first one. Will that workaround
> still work?

Uh... please elaborate. How would the resulting scene file look like?

Post a reply to this message

Am 23.05.2018 um 16:38 schrieb Stephen:
> On 23/05/2018 13:55, clipka wrote:
>> Am 23.05.2018 um 10:22 schrieb Stephen:
>>
>>> Test:
>>>
>>> v 3.8.0-alpha.9475849+av541.msvc14 is faster than
>>> v 3.8.0-x. tokenizer.9999+av5 60.msvc
>>>
>>> On the second run of the file. The parse time was a second or two slower
>>> than the first time. Except when using Ver 3.7 when it was the same.
>>
>> I take that as good news, presuming the scene in question does call
>> macros across files. Such a scenario would explain the superior
>> performance of v3.8.0-alpha over v3.7.0 as the result of macro caching,
>> as well as the inferior performance of v3.8.0-x.tokenizer vs.
>> v3.8.0-alpha as the result of disabled macro caching.
>>
> 
> Sorry it does not use macros. It did but I de-constructed the grass
> macro it was using as it was declaring an array of meshes from within a
> while loop. [Watch the memory usage rise and the HDD thrash.]

Hmm... then that begs the question, which change between v3.7.0 and
v3.8.0-alpha would have improved parsing speed?

Besides, what loop size are we talking about? Would that happen to be
>64k characters in the loop body?

> I have already had jr complain that I sent him the code in one file and
> not split into include files.
> So, what is wrong with 54317 lines in one file, anyway? ;-)

Nothing. It might actually be perfect for my parser tests. ;)

Post a reply to this message

On 23/05/2018 16:26, clipka wrote:
> Am 23.05.2018 um 16:18 schrieb Stephen:
> 
>>>> This is going to cause me problems using SDL generated by modellers.
>>>
>>> Care to do some name calling?
>>>
>>
>> Pal / Best friend. ;-)
> 
> Not exactly what I meant - I was rather trying to coax you into posting
> the name of the modeller, thereby calling it broken ;)
> 

Name and shame, then. No problem to me but you worked it out. Bishop3D

BTW As I keep saying to my wife, I don't do hints. You have to spell it 
out. Although not like her, in Morse code with a rubber mallet. ;)

3 dot, 2 dot 4 dot dash. ;)


> 
>>> Technically, those modellers would have to be considered broken:
>>
>> That seems fair enough.
>>
>>> To the
>>> best of my knowledge, single backslashes as filename separators have
>>> never been an officially supported feature.
>>>
>> By PovRay for Windows and/or for DOS?
> 
> Technically, that statement was true for all of them: I know precious
> little about POV-Ray for DOS, so to the best of my knowledge... ;)
> 
> 
> But I have to revert that statement now, having dug up an older
> documentation:
> 
> In POV-Ray 3.0, according to those docs, backslashes in string literals
> was only supposed to take on a special role when followed by a
> double-quote, in which case the backslash would be discarded while the
> double-quote would be interpreted as a character rather than the end of
> the string. Interpretation of other escape sequences was considered not
> a feature of string literals, but rather of user message stream output
> (`#debug` and its kin).
> 

There is a lot of history floating around. :)

> 
>> Significance is relevant. I have no idea how many people use B3D. And I
>> can't remember if Blender outputs the directory separator as a "\" or
>> "/". (I have stopped using it until some of the bug reports I have
>> submitted, have been fixed.)
> 
> Yeah, I guess B3D qualifies as relevant indeed.
> 
> 

Very pleased to hear it. :D

>> It is possible to get B3D to export bespoke code. So we could set a flag
>> telling the parser that the directory separator was "\".
>> e.g. windoze_separator = true
>> If that would help cut down the complexity.
> 
> Not sure I understand what you mean. Would you change B3D, or would you
> change POV-Ray?
> 
> When it comes to ideal format in POV-Ray scenes, it is highly
> recommended to always use "/" as path separator. Works on all platforms,
> 'Doze incldued.
> 

But not with MS-DOS 1.0. It didn’t support directories at all. The 
forward slash was used as a switch.

https://www.howtogeek.com/181774/why-windows-uses-backslashes-and-everything-else-uses-forward-slashes/


> 
>> Which reminds me. I use that facility to overwrite the version number
>> B3D exports by declaring it after the first one. Will that workaround
>> still work?
> 
> Uh... please elaborate. How would the resulting scene file look like?
> 

#version 3.6;


//------- Scene Raw Script Begin -------
#version 3.7 ;
//------- Scene Raw Script End ---------

[The rest of the scene...]

-- 

Regards
     Stephen

Post a reply to this message

On 23/05/2018 16:30, clipka wrote:
> Am 23.05.2018 um 16:38 schrieb Stephen:
>> On 23/05/2018 13:55, clipka wrote:
>>> Am 23.05.2018 um 10:22 schrieb Stephen:
>>>
>>>> Test:
>>>>
>>>> v 3.8.0-alpha.9475849+av541.msvc14 is faster than
>>>> v 3.8.0-x. tokenizer.9999+av5 60.msvc
>>>>
>>>> On the second run of the file. The parse time was a second or two slower
>>>> than the first time. Except when using Ver 3.7 when it was the same.
>>>
>>> I take that as good news, presuming the scene in question does call
>>> macros across files. Such a scenario would explain the superior
>>> performance of v3.8.0-alpha over v3.7.0 as the result of macro caching,
>>> as well as the inferior performance of v3.8.0-x.tokenizer vs.
>>> v3.8.0-alpha as the result of disabled macro caching.
>>>
>>
>> Sorry it does not use macros. It did but I de-constructed the grass
>> macro it was using as it was declaring an array of meshes from within a
>> while loop. [Watch the memory usage rise and the HDD thrash.]
> 
> Hmm... then that begs the question, which change between v3.7.0 and
> v3.8.0-alpha would have improved parsing speed?
> 

You mean it is not deliberate? o_O

> Besides, what loop size are we talking about? Would that happen to be
>> 64k characters in the loop body?
> 

I am not too sure I understand.

The loop size depends on the size of the target heightfield and the 
required density of the grass.

Relevant code.

             #declare BoxMin  = min_extent (Trace_patch_aa_) ;
             #declare BoxMax = max_extent (Trace_patch_aa_)  ;
             #declare Density =10000 ;  // User variable
             #declare RootDensity = sqrt(Density) ;




               #local Ox = BoxMin.x;
               #local Oz = BoxMin.z;
               #local Nx = int(RootDensity*sqrt(BoxMax.x - BoxMin.x));
               #local Nz = int(RootDensity*sqrt(BoxMax.z - BoxMin.z));
               #local Dx = Nx/Density;
               #local Dz = Nz/Density;

             union {
               #local I = 0;
               #while(I<Nx)
                #local J = 0;
                #while(J<Nz)
                 #local Z = Oz + ( J + 0.5 ) * Dz + ( rand( Pseed ) - 
0.5 ) * Dz * Pdev;
                 #local Z = ( ( Z < Oz ) ? Z - 2 * Oz : ( Z > -Oz ) ? Z 
+ 2 * Oz : Z );
                 #local X = Ox + ( I + 0.5 ) * Dx + ( rand( Pseed ) - 
0.5 ) * Dx * Pdev;
                 #local X = ( ( X < Ox ) ? X - 2 * Ox : ( X > -Ox ) ? X 
+ 2 * Ox : X );
                 #local W = 2 + ( rand( Wseed ) - 0.5 ) * Wdev;

                        #local Inter = trace ( Trace_object_ , < X , 
BoxMax.y + 100 , Z>,  < 0  ,  -1, 0 >, Norm);

                   #if (vlength (Norm)!=0)

USW


>> I have already had jr complain that I sent him the code in one file and
>> not split into include files.
>> So, what is wrong with 54317 lines in one file, anyway? ;-)
> 
> Nothing. It might actually be perfect for my parser tests. ;)
> 


I thought you might ask for a copy and started to look at how many image 
maps it was using.
It seems that when I was modifying the file manually I forgot to remove 
a couple of flower meshes. I had used in B3D. I cleaned it up and it is 
only 13500 lines now. And one heightfield image.

If you still want a copy. I will post it.
-- 

Regards
     Stephen

Post a reply to this message

On 23/05/2018 16:26, clipka wrote:
> Not sure I understand what you mean. Would you change B3D, or would you
> change POV-Ray?

Sorry I missed this.
B3D comes compiled with no source code. So for that to work it would 
need be a change in PovRay.

-- 

Regards
     Stephen

Post a reply to this message

Am 23.05.2018 um 18:53 schrieb Stephen:

>>> Which reminds me. I use that facility to overwrite the version number
>>> B3D exports by declaring it after the first one. Will that workaround
>>> still work?
>>
>> Uh... please elaborate. How would the resulting scene file look like?
>>
> 
> #version 3.6;
> 
> 
> //------- Scene Raw Script Begin -------
> #version 3.7 ;
> //------- Scene Raw Script End ---------
> 
> [The rest of the scene...]

If `#version 3.6;` is the very first statement in the scene, then that's
fine.(*)

If there's anything before that (other than whitespace and/or comments),
it is still fine unless you replace `#version 3.7;` with `#version 3.8;`.


(* In the sense that you won't get a parse error. It may have side
effects, e.g. the new v3.8 default values currently don't apply unless
you specify `#version 3.8` at the very first line. I intend to change
that though.)

Post a reply to this message

Am 23.05.2018 um 19:35 schrieb Stephen:
> On 23/05/2018 16:26, clipka wrote:
>> Not sure I understand what you mean. Would you change B3D, or would you
>> change POV-Ray?
> 
> Sorry I missed this.
> B3D comes compiled with no source code. So for that to work it would
> need be a change in PovRay.

Then such a switch would add more trouble than it would solve.

I'll try to re-implement v3.8.0-alpha behaviour, which is to provide
special treatment for filename literals if `#version 3.7` or earlier is
specified, while treating them like regular string literals if `#version
3.8` or later is specified.

In either case, a warning shall be printed if single backslashes are
encountered in filename literals.


("filename literal" being any string literal in a place where the parser
specifically expects a file name, such as in `#include`.)

Post a reply to this message

Am 23.05.2018 um 19:31 schrieb Stephen:

>>> Sorry it does not use macros. It did but I de-constructed the grass
>>> macro it was using as it was declaring an array of meshes from within a
>>> while loop. [Watch the memory usage rise and the HDD thrash.]
>>
>> Hmm... then that begs the question, which change between v3.7.0 and
>> v3.8.0-alpha would have improved parsing speed?
>>
> 
> You mean it is not deliberate? o_O

I don't recall having done anything to the parser between v3.7.0 and
v3.8.0-alpha with the intent to speed up parsing in general, with the
sole exception of macro caching. I hate the code, so it has never gotten
much love & care from me ;)

There /may/ have been side effects from a bit of code cleanup here and
there. But usually I've just given the parser more jobs to do.


>> Besides, what loop size are we talking about? Would that happen to be
>>> 64k characters in the loop body?
>>
> 
> I am not too sure I understand.
> 
> The loop size depends on the size of the target heightfield and the
> required density of the grass.

I mean the raw number of characters in the scene (or include) file, from
the `#while` or `#for` to the corresponding `#end`.

For example, the body of the following loop is 31 characters long:

    #for(I,0,100) #debug "Another iteration.\n" #end


> I thought you might ask for a copy and started to look at how many image
> maps it was using.
> It seems that when I was modifying the file manually I forgot to remove
> a couple of flower meshes. I had used in B3D. I cleaned it up and it is
> only 13500 lines now. And one heightfield image.
> 
> If you still want a copy. I will post it.

Please go ahead. I'm curious which feature of the file makes
v3.8.0-alpha like it better than v3.7.0.

Post a reply to this message

On 05/23/2018 02:05 PM, clipka wrote:
> Am 23.05.2018 um 19:31 schrieb Stephen:
> 
...
> 
> I don't recall having done anything to the parser between v3.7.0 and
> v3.8.0-alpha with the intent to speed up parsing in general, with the
> sole exception of macro caching. I hate the code, so it has never gotten
> much love & care from me ;)
> 
> There /may/ have been side effects from a bit of code cleanup here and
> there. But usually I've just given the parser more jobs to do.
> 
> 

There was somewhat recently this commit :

commit 26e2d47a35cebb91961642de0293d059b4fad071
Author: Christoph Lipka <c-l### [at] usersnoreplygithubcom>
Date:   Sat Jan 6 09:07:00 2018 +0100

     Improve parsing speed of skipped conditional blocks.

(some change from uberpov I think)

Bill P.

Post a reply to this message

On 05/23/2018 09:18 AM, clipka wrote:
> Am 22.05.2018 um 02:28 schrieb clipka:
...
> 
> The fact that none of you has reported any bugs yet makes me a bit
> uneasy... the new code can't be /that/ good ;)
> 

FYI. I've not run my personal set of parser tests against the new stuff. 
Takes time to turn and I'm working on other code (a.k.a. dazed and 
confused) at the moment.

Bill P.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Please give this version a thorough shakedown, with focus on parsing:
>

(running my original 'city buildings' code-- which is not very 'efficient',
although that might be a good thing for this test. With 1500 buildings.)

No #read's, no #include's, LOTS of #macros as simple containers (under 64K), of
this general form...
           #macro HO_16()
           jpeg "building windows 16.jpg" interpolate 2
           #end

.....which are used repeatedly, another *large* macro, two functions, LOTS of
comments, LOTS of image_maps, lots of #while loops.

v3.7.0: 35.54 seconds parse time

v3.8 tokenizer: 34.29 seconds

Hmm, too close to call (probably because of all those un-cached #macros, in the
tokenizer version.)

And no bugs! Sorry!! :-P

Post a reply to this message

Am 23.05.2018 um 20:37 schrieb William F Pokorny:

>> I don't recall having done anything to the parser between v3.7.0 and
>> v3.8.0-alpha with the intent to speed up parsing in general, with the
>> sole exception of macro caching. I hate the code, so it has never gotten
>> much love & care from me ;)
...

> There was somewhat recently this commit :
> 
> commit 26e2d47a35cebb91961642de0293d059b4fad071
> Author: Christoph Lipka <c-l### [at] usersnoreplygithubcom>
> Date:   Sat Jan 6 09:07:00 2018 +0100
> 
>     Improve parsing speed of skipped conditional blocks.

Dang - I guess I must love the parser more than I would ever publicly admit.

When I stumbled across the code again now for the overhaul, I somehow
was of the impression that it had been in there for ages.

Post a reply to this message

On 23/05/2018 19:05, clipka wrote:
> Am 23.05.2018 um 19:31 schrieb Stephen:
> 
>>>> Sorry it does not use macros. It did but I de-constructed the grass
>>>> macro it was using as it was declaring an array of meshes from within a
>>>> while loop. [Watch the memory usage rise and the HDD thrash.]
>>>
>>> Hmm... then that begs the question, which change between v3.7.0 and
>>> v3.8.0-alpha would have improved parsing speed?
>>>
>>
>> You mean it is not deliberate? o_O
> 
> I don't recall having done anything to the parser between v3.7.0 and
> v3.8.0-alpha with the intent to speed up parsing in general, with the
> sole exception of macro caching. I hate the code, so it has never gotten
> much love & care from me ;)
> 
> There /may/ have been side effects from a bit of code cleanup here and
> there. But usually I've just given the parser more jobs to do.
> 
> 
So Bill says. ;)


>>> Besides, what loop size are we talking about? Would that happen to be
>>>> 64k characters in the loop body?
>>>
>>
>> I am not too sure I understand.
>>
>> The loop size depends on the size of the target heightfield and the
>> required density of the grass.
> 
> I mean the raw number of characters in the scene (or include) file, from
> the `#while` or `#for` to the corresponding `#end`.
> 
> For example, the body of the following loop is 31 characters long:
> 
>      #for(I,0,100) #debug "Another iteration.\n" #end
> 

That is how I read it.
Including tabs. Word counts 399 characters

> 
>> I thought you might ask for a copy and started to look at how many image
>> maps it was using.
>> It seems that when I was modifying the file manually I forgot to remove
>> a couple of flower meshes. I had used in B3D. I cleaned it up and it is
>> only 13500 lines now. And one heightfield image.
>>
>> If you still want a copy. I will post it.
> 
> Please go ahead. I'm curious which feature of the file makes
> v3.8.0-alpha like it better than v3.7.0.
> 

Done.
Posted in povray.binaries.scene-files as "Flowers for clipka"

-- 

Regards
     Stephen

Post a reply to this message

New version:

https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9673626

Cached macros back in action.
Also includes an optimization that may reclaim some lost performance in
loops.

Post a reply to this message

On 23/05/2018 18:51, clipka wrote:
> Am 23.05.2018 um 18:53 schrieb Stephen:
> 
>>>> Which reminds me. I use that facility to overwrite the version number
>>>> B3D exports by declaring it after the first one. Will that workaround
>>>> still work?
>>>
>>> Uh... please elaborate. How would the resulting scene file look like?
>>>
>>
>> #version 3.6;
>>
>>
>> //------- Scene Raw Script Begin -------
>> #version 3.7 ;
>> //------- Scene Raw Script End ---------
>>
>> [The rest of the scene...]
> 
> If `#version 3.6;` is the very first statement in the scene, then that's
> fine.(*)
> 
> If there's anything before that (other than whitespace and/or comments),
> it is still fine unless you replace `#version 3.7;` with `#version 3.8;`.
> 
> 
> (* In the sense that you won't get a parse error. It may have side
> effects, e.g. the new v3.8 default values currently don't apply unless
> you specify `#version 3.8` at the very first line. I intend to change
> that though.)
> 

Thanks for clearing that up for me.
I seem to remember a warning that the version number must be the first 
statement and thought it might mean the final version number.

-- 

Regards
     Stephen

Post a reply to this message

On 23/05/2018 18:55, clipka wrote:
> Am 23.05.2018 um 19:35 schrieb Stephen:
>> On 23/05/2018 16:26, clipka wrote:
>>> Not sure I understand what you mean. Would you change B3D, or would you
>>> change POV-Ray?
>>
>> Sorry I missed this.
>> B3D comes compiled with no source code. So for that to work it would
>> need be a change in PovRay.
> 
> Then such a switch would add more trouble than it would solve.
> 

I must be over thinking. :)

> I'll try to re-implement v3.8.0-alpha behaviour, which is to provide
> special treatment for filename literals if `#version 3.7` or earlier is
> specified, while treating them like regular string literals if `#version
> 3.8` or later is specified.
> 

Great! That means that I can use 3.7 for testing and manually change the 
file when I want to use 3.8.



> In either case, a warning shall be printed if single backslashes are
> encountered in filename literals.
> 
> 
> ("filename literal" being any string literal in a place where the parser
> specifically expects a file name, such as in `#include`.)
> 

Another thought. (Don't judge me. ;) )
What about the library paths defined in povray.ini?

-- 

Regards
     Stephen

Post a reply to this message

Am 23.05.2018 um 22:16 schrieb Stephen:

> Another thought. (Don't judge me. ;) )
> What about the library paths defined in povray.ini?

Totally different story: Those don't go through the scanner/tokenizer,
so they are entirely unaffected by any change made to those parser
stages with regards to escape sequences.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> New version:
>
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9673626
>

Using my 'city buildings' scene again...

v3.7.0-- 35.54 seconds
v3.8 tokenizer-- 34.2 seconds (average of five runs)

(both versions running 'hot', as described earlier)

Post a reply to this message

Am 23.05.2018 um 21:55 schrieb Stephen:

>> Please go ahead. I'm curious which feature of the file makes
>> v3.8.0-alpha like it better than v3.7.0.
>>
> 
> Done.
> Posted in povray.binaries.scene-files as "Flowers for clipka"
> 

I think I nailed it: On my system I get just shy of 300 seconds parsing
time on v3.7.0, a bit over 200 seconds on v3.8.0-alpha, and a bit shy of
200 seconds on v3.8.0-x.tokenizer.9673626 now.

Didn't bother to test with v3.8.0-x.tokenizer.9999, as I would have had
to "rewind" to old source code, but I'm satisfied with having a
reasonably plausible theory as to why it may have been slower.

Post a reply to this message

On 23/05/2018 22:07, clipka wrote:
> Am 23.05.2018 um 21:55 schrieb Stephen:
> 
>>> Please go ahead. I'm curious which feature of the file makes
>>> v3.8.0-alpha like it better than v3.7.0.
>>>
>>
>> Done.
>> Posted in povray.binaries.scene-files as "Flowers for clipka"
>>
> 
> I think I nailed it: On my system I get just shy of 300 seconds parsing
> time on v3.7.0, a bit over 200 seconds on v3.8.0-alpha, and a bit shy of
> 200 seconds on v3.8.0-x.tokenizer.9673626 now.
> 
> Didn't bother to test with v3.8.0-x.tokenizer.9999, as I would have had
> to "rewind" to old source code, but I'm satisfied with having a
> reasonably plausible theory as to why it may have been slower.
> 


Excellent!
I'll download the new version you posted, tomorrow and rerun the scene.
BTW The scene I uploaded had twice the density of grass blades (10000) 
than the one I used originally to test them (5000).

-- 

Regards
     Stephen

Post a reply to this message

On 23/05/2018 21:33, clipka wrote:
> Am 23.05.2018 um 22:16 schrieb Stephen:
> 
>> Another thought. (Don't judge me. ;) )
>> What about the library paths defined in povray.ini?
> 
> Totally different story: Those don't go through the scanner/tokenizer,
> so they are entirely unaffected by any change made to those parser
> stages with regards to escape sequences.
> 

I imagine that would be the case. I was thinking more of consistency in 
the windoze environment.

-- 

Regards
     Stephen

Post a reply to this message

On 23-5-2018 14:20, clipka wrote:
> Am 23.05.2018 um 08:54 schrieb Thomas de Groot:
> 
>> Where the longer parse time comes from is in the while loop distributing
>> the cubes. This loop resides in a separate inc file. and looks like this:
>>
[snip]             ^^^^^^^^^^^^
> Unless you copied that to your file with the loop, that's a macro in a
> different file...
[snip]
> 
> So yes, that pretty much explains the poor performance with the
> overhauled parser -- or, more to the point, the better performance with
> v3.8.0-alpha: The latter features cached macros, while the overhauled
> parser currently has those disabled.
> 
> I would expect v3.7.0 to show roughly similar performance as the
> overhauled parser.
> 

Of course. I forgot about this. With version 3.7.0 the performance is as 
you say. Better performance starting with 3.7.1 according to my test.

-- 
Thomas

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> New version:
>
> https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9673626
>
.... and running Stephen's "Flowers For Clipka" scene. (I reduced his 'Density'
value from 10,000 down to 5,000, to be practical; and simply substituted his
image_map NAME in his height_field, instead of the library path.)

(Win7 64-bit, dual-core)

v3.7.0: 344 seconds parse (average of three runs)
v3.7.1 beta 9: 321 seconds (average of three runs)
v3.8 tokenizer: 225 seconds (average of three runs)

Nice! About a 35-percent decrease in parse time between 3.7.0 and 3.8-tokenizer,
very similar to Clipka's results.

Now I'm wondering why my 'city buildings' scene showed almost identical times
(3.7.0 vs. the 3.8 tokenizer), whereas Stephen's scene shows such a nice change.
The bulk of his scene is basically composed of four 'structures':

   mesh2 meshes
   trace(...)
   #while loops (three)
   height_field (one)

My own scene has only two of those four:
trace(...)  -- but not nearly as many trace 'rays' as Stephen's
#while loops

From this, it makes me wonder if the bulk of the parse-time-speedup is in mesh2
meshes and/or trace. Or maybe it isn't that simple to separate out where the
improvements originate from(?)

Post a reply to this message

On 23/05/2018 22:07, clipka wrote:
> Am 23.05.2018 um 21:55 schrieb Stephen:
> 
>>> Please go ahead. I'm curious which feature of the file makes
>>> v3.8.0-alpha like it better than v3.7.0.
>>>
>>
>> Done.
>> Posted in povray.binaries.scene-files as "Flowers for clipka"
>>
> 
> I think I nailed it: On my system I get just shy of 300 seconds parsing
> time on v3.7.0, a bit over 200 seconds on v3.8.0-alpha, and a bit shy of
> 200 seconds on v3.8.0-x.tokenizer.9673626 now.
> 
> Didn't bother to test with v3.8.0-x.tokenizer.9999, as I would have had
> to "rewind" to old source code, but I'm satisfied with having a
> reasonably plausible theory as to why it may have been slower.
> 

I ran the latest update to the tokenizer and here is a summary of the 
parse times.
Ver 3.7.0
   Parse Time:       0 hours  5 minutes 11 seconds (311.541 seconds)
Ver 3.8
   Parse Time:       0 hours  3 minutes 50 seconds (230.366 seconds)
Ver x.tokenizer.9999
   Parse Time:       0 hours  4 minutes 18 seconds (258.998 seconds)
Ver x.tokenizer.9673626
   Parse Time:       0 hours  3 minutes 19 seconds (199.541 seconds)


Aside.
I run Throttle.exe to monitor my CPU and GPU temperatures. I noticed 
that during the rendering of the scene. My CPU was being throttled back 
when I was using Ver 3.8 and the tokenizers. I cleaned my fans and 
filters last week so I don’t think it is an airflow problem.
Has anyone else noticed an increase in temperature with 3.8?


-- 

Regards
     Stephen

Post a reply to this message

On 24/05/2018 09:04, Kenneth wrote:
>  From this, it makes me wonder if the bulk of the parse-time-speedup is in mesh2
> meshes and/or trace. Or maybe it isn't that simple to separate out where the
> improvements originate from(?)

Before I cleaned up the scene for posting. There were two large mesh2 
opjects that were declared but not called. Using a density of 5000, 
there was no significient difference in parsing times.
I thought you might be onto something as the difference in Pov file size 
is 2230 KB to 596 KB.

-- 

Regards
     Stephen

Post a reply to this message

On 05/24/2018 05:25 AM, Stephen wrote:
> On 23/05/2018 22:07, clipka wrote:
...
> 
> Aside.
> I run Throttle.exe to monitor my CPU and GPU temperatures. I noticed 
> that during the rendering of the scene. My CPU was being throttled back 
> when I was using Ver 3.8 and the tokenizers. I cleaned my fans and 
> filters last week so I don’t think it is an airflow problem.
> Has anyone else noticed an increase in temperature with 3.8?
> 
> 

I've noted nothing with CPU temperature, but I don't routinely watch it. 
Do you see the throttling if you cut back on the number of cores you use?

What I am seeing in recent work is more volatility in performance 
measures than has been typical. In the range of -+0.5% run to run of 
late where I'd been getting <<0.1%. Makes it more difficult to determine 
whether a local code change has much affected performance. Basically 
need many more renders or renders with much longer run times.

Aside: Not talking about parser run times here. I expect those to be 
more volatile due memory allocations as the scene is digested.

Bill P.

Post a reply to this message

Am 24.05.2018 um 11:25 schrieb Stephen:

> Aside.
> I run Throttle.exe to monitor my CPU and GPU temperatures. I noticed
> that during the rendering of the scene. My CPU was being throttled back
> when I was using Ver 3.8 and the tokenizers. I cleaned my fans and
> filters last week so I don’t think it is an airflow problem.
> Has anyone else noticed an increase in temperature with 3.8?

Since I haven't touched the rendering itself, this must be a general
v3.8 thing.

Post a reply to this message

Am 24.05.2018 um 10:04 schrieb Kenneth:

> Now I'm wondering why my 'city buildings' scene showed almost identical times
> (3.7.0 vs. the 3.8 tokenizer), whereas Stephen's scene shows such a nice change.
> The bulk of his scene is basically composed of four 'structures':
> 
>    mesh2 meshes
>    trace(...)
>    #while loops (three)
>    height_field (one)
> 
> My own scene has only two of those four:
> trace(...)  -- but not nearly as many trace 'rays' as Stephen's
> #while loops
> 
> From this, it makes me wonder if the bulk of the parse-time-speedup is in mesh2
> meshes and/or trace. Or maybe it isn't that simple to separate out where the
> improvements originate from(?)

There is no speedup to be expected in the trace() function, nor is there
any speedup to be expected in mesh2 /per se/.

Just as a hunch, does your loop invoke any macros?

If so, it's fundamentally different from Stephen's loop: The latest
v3.8.0-x.tokenizer version can execute his loop from memory without
performing any file access during the loop, but only because it does not
invoke any macros.

Each return from macro currently causes the file buffer to be re-loaded.
To make matters worse, this re-loads only the portion /after/ the
location the macro returns to, so when the end of the loop is reached,
the next iteration requires another buffer re-load to get back to the
start of the loop.

Post a reply to this message

On 24/05/2018 12:13, William F Pokorny wrote:
> On 05/24/2018 05:25 AM, Stephen wrote:
>> On 23/05/2018 22:07, clipka wrote:
> ...
>>
>> Aside.
>> I run Throttle.exe to monitor my CPU and GPU temperatures. I noticed 
>> that during the rendering of the scene. My CPU was being throttled 
>> back when I was using Ver 3.8 and the tokenizers. I cleaned my fans 
>> and filters last week so I don’t think it is an airflow problem.
>> Has anyone else noticed an increase in temperature with 3.8?
>>
>>
> 
> I've noted nothing with CPU temperature, but I don't routinely watch it. 

For years I ran Pov on a laptop. When your lap starts to burn it makes 
you think. Now it is just a habit.

> Do you see the throttling if you cut back on the number of cores you use?
> 
Looking at the temp graph. I see that one core is running about 5°C 
above the lowest. And yes if I deny that core to Pov then it does not 
throttle. I have set the throttling level at 89°C that is 6°C below the 
CPU shutdown level*. There is a distinct difference between 3.7 and 3.8.

Ach well, it will give me something to do in my free time. Like moving 
the box away from a glass partition.


*
If the internet can be believed. There are a lot of conflicting opinions 
(crap) out there. Obviously no one on the over-clocker forums uses Pov. :)


> What I am seeing in recent work is more volatility in performance 
> measures than has been typical. In the range of -+0.5% run to run of 
> late where I'd been getting <<0.1%. Makes it more difficult to determine 
> whether a local code change has much affected performance. Basically 
> need many more renders or renders with much longer run times.
> 
> Aside: Not talking about parser run times here. I expect those to be 
> more volatile due memory allocations as the scene is digested.
> 

That is a bit above my pay grade. ;)

-- 

Regards
     Stephen

Post a reply to this message

On 24/05/2018 12:42, clipka wrote:
> Am 24.05.2018 um 11:25 schrieb Stephen:
> 
>> Aside.
>> I run Throttle.exe to monitor my CPU and GPU temperatures. I noticed
>> that during the rendering of the scene. My CPU was being throttled back
>> when I was using Ver 3.8 and the tokenizers. I cleaned my fans and
>> filters last week so I don’t think it is an airflow problem.
>> Has anyone else noticed an increase in temperature with 3.8?
> 
> Since I haven't touched the rendering itself, this must be a general
> v3.8 thing.
> 

Understood, you are working on the parser. That is why I said "aside" 
and mentioned it in case it might have some significance for something.


-- 

Regards
     Stephen

Post a reply to this message

Am 24.05.2018 um 14:19 schrieb Stephen:

> Looking at the temp graph. I see that one core is running about 5°C
> above the lowest. And yes if I deny that core to Pov then it does not
> throttle. I have set the throttling level at 89°C that is 6°C below the
> CPU shutdown level*. There is a distinct difference between 3.7 and 3.8.

If by "3.7" you mean v3.7.0, then a potential reason for such a
difference might arise from the build tools used: While POV-Ray v3.7.0
was built using Visual Studio 2010, more recent versions are built using
Visual Studio 2015. Differences in how the compilers optimize the code
may well result in differences in how heavily the CPU can be utilized.

In a similar vein, the hand-optimized noise generator implementations
may also play a role: While v3.7.0 could only utilize AVX/FMA4 for this
purpose, newer versions can also utilize AVX2/FMA3 as well as "pure" AVX
(and the AVX/FMA4 version has been improved).


TL;DU: There are plausible explanations why current versions of POV-Ray
may be utilizing your CPU more heavily (or at least differently) than
v3.7.0 did.

Post a reply to this message

Am 23.05.2018 um 10:22 schrieb Stephen:
> On 22/05/2018 01:28, clipka wrote:
>> - Backward compatibility with scenes that use single backslashes in
>> literal filenames has been sacrificed, and will most likely not be
>> restored.
> 
> This is going to cause me problems using SDL generated by modellers.

I'm zeroing in on the conclusion that this is a problem that's not
easily solved in the new parser architecture, especially with respect to
the future plans.


The aim is to scan/tokenize a file only once, cache the result, and from
then on only operate on the resulting token stream. Maybe even have a
separate scanner/tokenizer thread go ahead with its job while the parser
thread is still busy with earlier portions of the scene.

This means that whatever looks identical, must be scanned/tokenized
identically. Most notably, backslashes in string literals must be
treated consistently: "foo\nbar" must always be translated as the
character sequence `foo` followed by a newline followed by the character
sequence `bar`, no matter whether the parser - based on neighboring
tokens - expects a filename or a generic string at that particular location.

-OR- the scanner/tokenizer would always have to translate such a string
literal as the character sequence `foo\nbar`. But then the job of
resolving the escape sequence would fall to the parser proper, and would
have to be done over and over again each time the parser encounters the
string literal. Since strings are comparatively heavyweight to process,
that would be bad news in terms of performance.


A similar problem exists with respect to character encoding: Without
understanding of `global_settings { charset utf8 }`, the
scanner/tokenizer cannot decide whether a non-ASCII byte sequence in a
string literal should be interpreted according to UTF-8 or according to
Windows-1252, Latin-1 or whatever.

Post a reply to this message

On 24/05/2018 15:54, clipka wrote:
> Am 23.05.2018 um 10:22 schrieb Stephen:
>> On 22/05/2018 01:28, clipka wrote:
>>> - Backward compatibility with scenes that use single backslashes in
>>> literal filenames has been sacrificed, and will most likely not be
>>> restored.
>>
>> This is going to cause me problems using SDL generated by modellers.
> 
> I'm zeroing in on the conclusion that this is a problem that's not
> easily solved in the new parser architecture, especially with respect to
> the future plans.
> 
> 
> The aim is to scan/tokenize a file only once, cache the result, and from
> then on only operate on the resulting token stream. Maybe even have a
> separate scanner/tokenizer thread go ahead with its job while the parser
> thread is still busy with earlier portions of the scene.
> 
> This means that whatever looks identical, must be scanned/tokenized
> identically. Most notably, backslashes in string literals must be
> treated consistently: "foo\nbar" must always be translated as the
> character sequence `foo` followed by a newline followed by the character
> sequence `bar`, no matter whether the parser - based on neighboring
> tokens - expects a filename or a generic string at that particular location.
> 
> -OR- the scanner/tokenizer would always have to translate such a string
> literal as the character sequence `foo\nbar`. But then the job of
> resolving the escape sequence would fall to the parser proper, and would
> have to be done over and over again each time the parser encounters the
> string literal. Since strings are comparatively heavyweight to process,
> that would be bad news in terms of performance.
> 
> 
> A similar problem exists with respect to character encoding: Without
> understanding of `global_settings { charset utf8 }`, the
> scanner/tokenizer cannot decide whether a non-ASCII byte sequence in a
> string literal should be interpreted according to UTF-8 or according to
> Windows-1252, Latin-1 or whatever.
> 

Don't give me your problems. I have enough of my own. :)

A solution would be to get started on the replacement for Moray. The one 
that Lutz Kretzschmar gave the licence, to the Pov group, in Feb 2007.
http://www.stmuc.com/moray/menews.html
[Don't say that I am not patient.]

-- 

Regards
     Stephen

Post a reply to this message