I've borrowed some code Bill Walker (Bald Eagle) posted in the 
"Isosurface and f_spiral - bug or feature?"  thread started by Ronkar to 
demonstrate a couple of parser performance issues/user-optimizations.

1) Comments in inner loops are costly. 2) Functions are often much 
faster than macros.

With both, parser performance can be positively affected by how we code. 
  Perhaps, the parser code itself can better handle comments. Details 
and sample code are part of my comment today to the github issue:

https://github.com/POV-Ray/povray/issues/313

Bill P.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:
> I've borrowed some code Bill Walker (Bald Eagle) posted in the
> "Isosurface and f_spiral - bug or feature?"  thread started by Ronkar to
> demonstrate a couple of parser performance issues/user-optimizations.
>
> 1) Comments in inner loops are costly. 2) Functions are often much
> faster than macros.

I would not have guessed that.  I had always imagined that it bailed out as soon
as it saw //  but I guess not.
Also good to know if
Code /* inner comment */  more code... is used in a scene file

> With both, parser performance can be positively affected by how we code.

Yes, I try to learn and apply good coding practices as often as possible - which
sometimes isn't that often.
I had been trying to query why a polynomial is [supposed to be] much faster than
an isosurface or parametric - I have ideas, but no "source code explanation"

>   Perhaps, the parser code itself can better handle comments. Details
> and sample code are part of my comment today to the github issue:
>
> https://github.com/POV-Ray/povray/issues/313
>
> Bill P.


BTW,
Using 3.7.1-beta.8

#version 3.71;
#declare StartTime = now();

Gave me a "Need a semicolon" errors.
Bug Report?

Post a reply to this message

Le 17-10-07 à 19:00, Bald Eagle a écrit :
> William F Pokorny <ano### [at] anonymousorg> wrote:
>> I've borrowed some code Bill Walker (Bald Eagle) posted in the
>> "Isosurface and f_spiral - bug or feature?"  thread started by Ronkar to
>> demonstrate a couple of parser performance issues/user-optimizations.
>>
>> 1) Comments in inner loops are costly. 2) Functions are often much
>> faster than macros.
> 
> I would not have guessed that.  I had always imagined that it bailed out as soon
> as it saw //  but I guess not.

It does ignore whatever is in the coment, but, it have to pace through 
the comment character by character to find exactly where it ends. In a 
loop, it add up quickly.

> Also good to know if
> Code /* inner comment */  more code... is used in a scene file
Same thing. Need to read it all to find the end point.

So, put as much of your coment before the loops so that the parser only 
have to skip it once.

> 
>> With both, parser performance can be positively affected by how we code.
> 
> Yes, I try to learn and apply good coding practices as often as possible - which
> sometimes isn't that often.
> I had been trying to query why a polynomial is [supposed to be] much faster than
> an isosurface or parametric - I have ideas, but no "source code explanation"
> 
>>    Perhaps, the parser code itself can better handle comments. Details
>> and sample code are part of my comment today to the github issue:
>>
>> https://github.com/POV-Ray/povray/issues/313
>>
>> Bill P.
> 
> 
> BTW,
> Using 3.7.1-beta.8
> 
> #version 3.71;
> #declare StartTime = now();
> 
> Gave me a "Need a semicolon" errors.
> Bug Report?
> 
>

Post a reply to this message

Am 08.10.2017 um 01:00 schrieb Bald Eagle:

> #version 3.71;
> #declare StartTime = now();
> 
> Gave me a "Need a semicolon" errors.
> Bug Report?

`now` does not act as a function, but rather as a pseudo-variable. Use

    #declare StartTime = now;

and you should be fine.

Post a reply to this message

On 10/06/2017 01:18 PM, William F Pokorny wrote:
> Details and sample code are part of my comment today to the github issue:
> 
> https://github.com/POV-Ray/povray/issues/313
> 
> Bill P.

Christoph asked me to limit my comments on github to those specific to 
the feature proposal in 313 so moving the bulk of the text posted there 
below so it is not lost and can still be referred to on github.

--- Moved text below ---

I've borrowed code Bill Walker (Bald Eagle) posted in the "Isosurface 
and f_spiral - bug or feature?"  thread started by Ronkar:

http://news.povray.org/povray.newusers/thread/%3Cweb.59d6bdad808cf4025cafe28e0%40news.povray.org%3E/

to demonstrate a couple of parser performance issues/user-optimizations. 
Both of which can be positively affected by how we code.  One of which, 
perhaps, the parser code itself can better handle.

First, writing macros partly or wholly as functions can quite often 
provide a significant parser speed up.  In the code below 10.78sec to 
3.31sec, or -69%, for Bill's code with further optimization still possible.

I think about our include files - CRGB2H in colors.inc for example - and 
I wonder how much parser performance improvement we'd get by re-coding 
to functions.  Or perhaps adding function equivalents to the current 
macros as the functional equivalents are less easy to understand on 
quick viewing.  I have rattling around in my head someone (Trevor 
Quayle?)  already coded many color space 'functions' as functions.  I 
wasn't able to quickly find the work just now and with the pigment { 
user_defined { } } feature of 3.8 maybe the approach would today be 
different...

In using functions we use our VM which already pre-calculates everything 
it can.  In other words in using functions we take the parsing / 
re-parsing / re-calculating which happens with macros and turn it into 
VM 'compiled' code which can be called.

Second, comments in inner loops are costly. I believe, but did not 
prove, the comments in Bill's Spiral macro are some part of why the 
functional version is faster in the code below.  To demonstrate the 
comment effect more plainly, the code below has a block of comments 
which, on copy/move into the inner-most for loop, slow the parsing by 
20+% and 60+% depending on whether using the original Spiral macro or 
the FN0to1Val function set up by a once called macro.

Users can move comments outside the loops/macros and help themselves, or 
pre-process SDL code to remove all comments themselves. However,  here I 
have to think the parser itself can do better.  FYI - Just shortening 
the block of comments to leave only the leading '//' characters speeds 
the parser up considerably.  Might the parser bail on comment lines 
earlier and/or on first read create a simpler representation for 
comments while maintaining the line counts?  I've done no digging in the 
parser code yet myself with these ideas in mind.

-----------------------------
Code below run and timed on Ubuntu linux with the command:

/usr/bin/time povray the.pov


//--- Equivalent coding using macro defining function
//    for faster parsing. Demo slow comments. WFP Oct 6, 2017
#version 3.7;
//------------------------------------------
// SDL for spiral distance field
// Bill Walker - October 2017
//------------------------------------------
global_settings {assumed_gamma 1}

#include "colors.inc"
#include "consts.inc"

light_source { <-1,8,2> color White}

camera {
  orthographic
  location <0, 0, -100>
  look_at  <0, 0,   0>
  right x*image_width
  up y*image_height
}

plane {z, 1 pigment {checker Gray20, White} scale 10}
light_source {<0, 10, -500> color White*0.5}

//----- This all macro method OK, but about 225% slower than
//      than using a single macro call to create a function.
#macro Spiral (X, Y)
  #declare a=1;
  #declare b=0.5;
  // calculate the target radius and theta
   #local R = sqrt (X*X +Y*Y);

    // early exit if the point requested is the origin itself
  // to avoid taking the logarithm of zero in the next step
  #if (R = 0)
   0
  #else
   #local T = atan2 (Y, X);
   // calculate the floating point approximation for n
   #local n = (log(R/a)/b - T)/(tau);

   // find the two possible radii for the closest point
   #local upper_r = a * pow(e, b * (T + tau*ceil(n)));
   #local lower_r = a * pow(e, b * (T + tau*floor(n)));

   // return the minimum distance to the target point
   min (abs(upper_r - R), abs(R - lower_r))
  #end
#end

// - Yes this a fairly straight map and it could be further optimized.
#macro DefFN0to1Val(A,B)
     #declare FNR  = function (x,y,z,X,Y) { sqrt(X*X+Y*Y) }
     #declare FNT  = function (x,y,z,X,Y) { atan2(Y,X) }
     #declare FNn  = function (x,y,z,X,Y) {
         (log(FNR(x,y,z,X,Y)/A)/B - FNT(x,y,z,X,Y))/tau
     }
     #declare FNuR = function (x,y,z,X,Y) {
         A * pow(e,B * (FNT(x,y,z,X,Y) + tau*ceil(FNn(x,y,z,X,Y))))
     }
     #declare FNlR = function (x,y,z,X,Y) {
         A * pow(e,B * (FNT(x,y,z,X,Y) + tau*floor(FNn(x,y,z,X,Y))))
     }
     #declare FNspiral = function (x,y,z,X,Y) {
         min(abs(FNuR(x,y,z,X,Y)-FNR(x,y,z,X,Y)),
             abs(FNR(x,y,z,X,Y)-FNlR(x,y,z,X,Y))
         )
     }
     #declare FN0to1Val = function (x,y,z,X,Y) {
         min(1,FNspiral(x,y,z,X,Y)/255)
     }
#end
DefFN0to1Val(1.0,0.5)

// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
// Adding this comment to most nested for loop 10 times is >20% slower.
#for (i, -image_width/2, image_width/2)
  #for (j, -image_height/2, image_height/2)
    box { -0.5,0.5 translate <i,j,0>
          pigment {color rgb FN0to1Val(0,0,0,i,j)}
        }
  #end
#end

// Options are NOT commented / un-commented in the nested for loops
// above because longish comments there cause a parser slow down.
//
//---  (slower) 10.80user 0.15system 0:11.50elapsed
// box { -0.5,0.5 translate <i,j,0>
//       pigment {color rgb min(1, Spiral(i,j)/255)}
//     }
//---  (faster) 3.41user 0.17system 0:04.13elapsed
// box { -0.5,0.5 translate <i,j,0>
//       pigment {color rgb FN0to1Val(0,0,0,i,j)}
//     }

#error "Stop after parse for time comparison"

Post a reply to this message

Very nice.
Thanks for noticing this issue - I'll have to re-think the way I comment my
code.

I rewrote your scene to do the timing in SDL and compare the elapsed times for
each method.
For large scenes with many slow objects and loops, this may help people
investigate "Why is my scene SO SLOW?"

This once again (although in a different way) highlights the macro vs function
approach discussed in the documentation.






//--- Equivalent coding using macro defining function
//    for faster parsing. Demo slow comments. WFP Oct 6, 2017
//  Edited to record elapsed times in SDL using "now" directive
//  and compare for different methods
#version 3.7;
//------------------------------------------
// SDL for spiral distance field
// Bill Walker - October 2017
//------------------------------------------
global_settings {assumed_gamma 1}
#include "colors.inc"
#include "consts.inc"
light_source { <-1,8,2> color White}
camera {
 orthographic
 location <0, 0, -100>
 look_at  <0, 0,   0>
 right x*image_width
 up y*image_height
}
plane {z, 1 pigment {checker Gray20, White} scale 10}
light_source {<0, 10, -500> color White*0.5}

//----- This all macro method OK, but about 225% slower than
//      than using a single macro call to create a function.
//  BW - I get 3.184 times longer than the function-only version
#macro Spiral (X, Y)
  #declare a=1;
  #declare b=0.5;
  // calculate the target radius and theta
   #local R = sqrt (X*X +Y*Y);

    // early exit if the point requested is the origin itself
  // to avoid taking the logarithm of zero in the next step
  #if (R = 0)
   0
  #else
   #local T = atan2 (Y, X);
   // calculate the floating point approximation for n
   #local n = (log(R/a)/b - T)/(tau);

   // find the two possible radii for the closest point
   #local upper_r = a * pow(e, b * (T + tau*ceil(n)));
   #local lower_r = a * pow(e, b * (T + tau*floor(n)));

   // return the minimum distance to the target point
   min (abs(upper_r - R), abs(R - lower_r))
  #end
#end



// - Yes this a fairly straight map and it could be further optimized.
#macro DefFN0to1Val(A,B)
     #declare FNR  = function (x,y,z,X,Y) { sqrt(X*X+Y*Y) }
     #declare FNT  = function (x,y,z,X,Y) { atan2(Y,X) }
     #declare FNn  = function (x,y,z,X,Y) {
         (log(FNR(x,y,z,X,Y)/A)/B - FNT(x,y,z,X,Y))/tau
     }
     #declare FNuR = function (x,y,z,X,Y) {
         A * pow(e,B * (FNT(x,y,z,X,Y) + tau*ceil(FNn(x,y,z,X,Y))))
     }
     #declare FNlR = function (x,y,z,X,Y) {
         A * pow(e,B * (FNT(x,y,z,X,Y) + tau*floor(FNn(x,y,z,X,Y))))
     }
     #declare FNspiral = function (x,y,z,X,Y) {
         min(abs(FNuR(x,y,z,X,Y)-FNR(x,y,z,X,Y)),
             abs(FNR(x,y,z,X,Y)-FNlR(x,y,z,X,Y))
         )
     }
     #declare FN0to1Val = function (x,y,z,X,Y) {
         min(1,FNspiral(x,y,z,X,Y)/255)
     }
#end

#macro UndefAll ()
// Undefine functions so that function-defining macro can be called again
     #undef FNR
     #undef FNT
     #undef FNn
     #undef FNuR
     #undef FNlR
     #undef FNspiral
     #undef FN0to1Val
#end

//##########################################################################################
#declare Seconds = 24*60*60;


//##########################################################################################
//  Original code adapted to SDL from python source at:
//
https://swiftcoder.wordpress.com/2010/06/21/logarithmic-spiral-distance-field/
//##########################################################################################
#declare StartTime = now*Seconds;
#debug concat ( "Beginning timer. Time elapsed = ", str (StartTime-StartTime, 4,
3),  " sec. \n")
#for (i, -image_width/2, image_width/2)
 #for (j, -image_height/2, image_height/2)
  box { -0.5, 0.5 translate <i, j, 0> pigment {color rgb min(1, Spiral(i, j)/255
)} }
 #end
#end
#declare CurrentTime = now*Seconds;
#declare ElapsedTime_1 = CurrentTime-StartTime;
#debug concat ( "Original macro-in-nested-loop method. Time elapsed = ", str
(ElapsedTime_1, 4, 3),  " sec. \n")


//##########################################################################################
//  SDL using functions (defined by calling a macro ONCE) by William Pokorny:
//
http://news.povray.org/povray.general/message/%3C59da1882%40news.povray.org%3E/#%3C59da1882%40news.povray.org%3E
//##########################################################################################
#declare StartTime = now*Seconds;
#debug concat ( "Beginning timer. Time elapsed = ", str (StartTime-StartTime, 4,
3),  " sec. \n")
DefFN0to1Val(1.0,0.5)
#for (i, -image_width/2, image_width/2)
  #for (j, -image_height/2, image_height/2)
    box { -0.5,0.5 translate <i,j,0>
          pigment {color rgb FN0to1Val(0,0,0,i,j)}
        }
  #end
#end
#declare CurrentTime = now*Seconds;
#declare ElapsedTime_2 = CurrentTime-StartTime;
#debug concat ( "WP's Function-in-nested-loop method. Time elapsed = ", str
(ElapsedTime_2, 4, 3),  " sec. \n")


//##########################################################################################
//  SDL using functions (defined by calling a macro ONCE) by William Pokorny:
//
http://news.povray.org/povray.general/message/%3C59da1882%40news.povray.org%3E/#%3C59da1882%40news.povray.org%3E
//  COMMENTS PRESENT IN INNER NESTED LOOP
//  BW - I get 1.571 times longer than the function-only version
//##########################################################################################
UndefAll () // Undefine functions so that function-defining macro can be called
again in this block for accurate comparison
#declare StartTime = now*Seconds;
#debug concat ( "Beginning timer. Time elapsed = ", str (StartTime-StartTime, 4,
3),  " sec. \n")
DefFN0to1Val(1.0,0.5)
#for (i, -image_width/2, image_width/2)
  #for (j, -image_height/2, image_height/2)
    box { -0.5,0.5 translate <i,j,0>
          pigment {color rgb FN0to1Val(0,0,0,i,j)}
        }
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  // Adding this comment to most nested for loop 10 times is >20% slower.
  #end
#end
#declare CurrentTime = now*Seconds;
#declare ElapsedTime_3 = CurrentTime-StartTime;
#debug concat ( "WP's Function-in-nested-loop method (with comments). Time
elapsed = ", str (ElapsedTime_3, 4, 3),  " sec. \n")
#debug "\n"
#debug "\n"
#debug concat ( "   Macro-in-nested-loop method.                 Time elapsed =
", str (ElapsedTime_1, 4, 3),  " sec. ", str (ElapsedTime_1/ElapsedTime_2, 4,
3), " times slower. \n")
#debug concat ( "Function-in-nested-loop method.                 Time elapsed =
", str (ElapsedTime_2, 4, 3),  " sec. ", str (ElapsedTime_2/ElapsedTime_2, 4,
3), " [base time]. \n")
#debug concat ( "Function-in-nested-loop method (with comments). Time elapsed =
", str (ElapsedTime_3, 4, 3),  " sec. ", str (ElapsedTime_3/ElapsedTime_2, 4,
3), " times slower. \n")
#debug "\n"
#error "Aborting render phase.  Stopped after parse for time comparison of
coding methods."

Post a reply to this message

Alain <kua### [at] videotronca> wrote:
> Le 17-10-07 à 19:00, Bald Eagle a écrit :
> > William F Pokorny <ano### [at] anonymousorg> wrote:
> >>
> >> 1) Comments in inner loops are costly.
> >
> > I would not have guessed that.  I had always imagined that it bailed out
> > as soon as it saw  //  but I guess not.
>
> It does ignore whatever is in the comment, but, it have to pace through
> the comment character by character to find exactly where it ends. In a
> loop, it add up quickly.
>

Thanks, Alain. That explains my own longer parse times when using lots of
comments in a scene, the reason for which has been a mystery to me for years--
because the documentation makes a point of saying "comments are ignored by the
raytracer" and "Use comments liberally and generously." ;-)

My own (false) understanding was that a  // or  /* comment indicator somehow
caused the parser to actually *skip* all of the following text, or until it hit
another 'closing' */ indicator-- kind of like a  'goto'  statement in other
languages, or like POV's macro-invocation behavior, where the parser immediately
jumps to the macro itself. But it makes sense that the parser has to 'read' the
intervening text-- at least when using  /*  and  */ --in order to know WHERE the
closing comment is! For single-line  // comments, though, I would have imagined
that all the text on that particular line would BE ignored, as soon as the
parser came across a double slash //  Hmm, apparently not.

I need to look through my scenes to try and eliminate as many unnecessary
comments as possible in any #while/#for loops, and in any #macros that are
called multiple times.

Post a reply to this message

"Kenneth" <kdw### [at] gmailcom> wrote:
For single-line  // comments, though, I would have imagined
> that all the text on that particular line would BE ignored, as soon as the
> parser came across a double slash //  Hmm, apparently not.

I think I initially had that impression, but if you think about it, the parser
has to look for a NL/CR line ENDING character so that it knows what the next
line is.

Why this is so SLOW for the parser to do remains puzzling to me, since I can
download hundreds of megabytes, use FIND to search 1000+ page PDF's for text, or
search a whole directory or drive to find that text, and I have a positively
insane spreadsheet I wrote at work which does I don't even want to think about
how many _calculations_ .....

Maybe it's just old, horribly outdated / outmoded coding of the parser.

<Gollum> Clipka hates the parser, yes he does, my Precioussssssss  <Gollum>

Post a reply to this message

Am 11.10.2017 um 11:44 schrieb Kenneth:

> My own (false) understanding was that a  // or  /* comment indicator somehow
> caused the parser to actually *skip* all of the following text, or until it hit
> another 'closing' */ indicator-- kind of like a  'goto'  statement in other
> languages, or like POV's macro-invocation behavior, where the parser immediately
> jumps to the macro itself. But it makes sense that the parser has to 'read' the
> intervening text-- at least when using  /*  and  */ --in order to know WHERE the
> closing comment is! For single-line  // comments, though, I would have imagined
> that all the text on that particular line would BE ignored, as soon as the
> parser came across a double slash //  Hmm, apparently not.

It does make sense when you think of how the individual lines in an
ASCII file are actually stored.

There is no "length of line" or "pointer to end of line" stored in an
ASCII file; instead, a special character (0x0A = LF = Line Feed; Unix
convention) or sequence thereof (0x0D,0x0A = CR,LF = Carriage Return,
Line Feed; DOS/Windows convention) is used to mark the end of a line.

So even with single-line comments the parser needs to scan the text
character by character to find the end of the comment.

Post a reply to this message

Am 11.10.2017 um 12:31 schrieb Bald Eagle:
> "Kenneth" <kdw### [at] gmailcom> wrote:
> For single-line  // comments, though, I would have imagined
>> that all the text on that particular line would BE ignored, as soon as the
>> parser came across a double slash //  Hmm, apparently not.
> 
> I think I initially had that impression, but if you think about it, the parser
> has to look for a NL/CR line ENDING character so that it knows what the next
> line is.
> 
> Why this is so SLOW for the parser to do remains puzzling to me, since I can
> download hundreds of megabytes, use FIND to search 1000+ page PDF's for text, or
> search a whole directory or drive to find that text, and I have a positively
> insane spreadsheet I wrote at work which does I don't even want to think about
> how many _calculations_ .....
> 
> Maybe it's just old, horribly outdated / outmoded coding of the parser.

A search is a simple thing: You only traverse each document once, and
you're done.

A search in a directory is even easier: On a modern computer with
standard settings, you actually only search an index.

Parsing a loop... well, it /could/ be reasonably easy if the parser
pre-processed the content somehow (e.g. discarding any comments,
identifying keywords, and some such), and on each loop iteration only
traversed the pre-processed stuff. Unfortunately, POV-Ray's organically
grown syntax is quite detrimental to such an approach.


> <Gollum> Clipka hates the parser, yes he does, my Precioussssssss  <Gollum>

Oh yes, we does, doesn't we, my precioussss? Oh yes, we loathes it!

It's not tasty... it's not scrumptious... no, preciousssss! Not at all
scrumptious! Nassssty parsers!

Post a reply to this message