On 4/26/21 3:59 PM, jr wrote:
> "jr" <cre### [at] gmailcom> wrote:
>> posting #4 of 'Foreach()'...
> 
> tried to see how many levels of nested calls of 'Foreach()' POV-Ray will do,
> given that the documentation[*] does not mention a limit.  got to nine.  when I
> uncomment the line in 'm_l9', the error shown below code (attached) occurs.
> this is the "call stack" getting trashed?
> 

Short on time today, but maybe something with turning and starting back 
up the macro stack? The parse_fore.tmp file looks to contain the last 
m_l10 'execution string'. Is it somehow getting run an extra time on the 
ascent?

If I change m_l8 to call m_l10, I get the same error and it happens 
after all four calls to m_l10 at a depth of 8.

Aside: I don't think any v3.7 version will work to debug this as the 
dictionary feature is new to v3.8.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 4/26/21 3:59 PM, jr wrote:
> > tried to see how many levels of nested calls of 'Foreach()' POV-Ray ...
>
> Short on time today, but maybe something with turning and starting back
> up the macro stack? The parse_fore.tmp file looks to contain the last
> m_l10 'execution string'. Is it somehow getting run an extra time on the
> ascent?

thanks for the quick look.  (I don't know the code base)  all foreach calls use
the only array in scene.  no idea where an extra run/invocation [cw]ould come
from.


> If I change m_l8 to call m_l10, I get the same error and it happens
> after all four calls to m_l10 at a depth of 8.

neat.  never even occurred to me.  the 'nesting.pov' "evolved", copy+paste
style, by level, so the "last" macro always was just a '#debug concat', to tie
in with above.

the output below was done with an alpha with three of your patches, plus
clipka's to get it to build.  just .. strange.


> Aside: I don't think any v3.7 version will work to debug this as the
> dictionary feature is new to v3.8.

interestingly, after rigging version and removing the 'global' in line 73 (I
think), the parser does not object to the .inc, in spite of my using 'dict.key'
notation throughout.  may have a play with this Debian build (todo list. :-))


regards, jr.


Script started on Tue 27 Apr 2021 11:23:30 BST

jr@swift:1:foreach$ tail -n15 nesting.pov
#end

#macro m_l2(i_,n_)
  #debug concat("--> level 2, n = ",str(n_,0,0),".\n")
//  Foreach(A, dictionary {.Macro: "m_l3"})
//  Foreach(A, dictionary {.Macro: "m_l10"})
#end

#macro m_l1(i_,n_)
  #debug concat("-> level 1, n = ",str(n_,0,0),".\n")
  Foreach(A, dictionary {.Macro: "m_l2"})
#end

Foreach(A, dictionary {.Macro: "m_l1"})


jr@swift:2:foreach$ povparse nesting.pov
Persistence of Vision(tm) Ray Tracer Version 3.8.0-alpha.10064268.unofficial
  ...
==== [Parsing...] ==========================================================
-> level 1, n = 1.
--> level 2, n = 1.
--> level 2, n = 2.
--> level 2, n = 3.
--> level 2, n = 4.
-> level 1, n = 2.
--> level 2, n = 1.
--> level 2, n = 2.
--> level 2, n = 3.
--> level 2, n = 4.
-> level 1, n = 3.
--> level 2, n = 1.
--> level 2, n = 2.
--> level 2, n = 3.
--> level 2, n = 4.
-> level 1, n = 4.
--> level 2, n = 1.
--> level 2, n = 2.
--> level 2, n = 3.
--> level 2, n = 4.
File 'nesting.pov' line 327: Parse Warning: No objects in scene.


jr@swift:3:foreach$ tail -n15 nesting.pov
#end

#macro m_l2(i_,n_)
  #debug concat("--> level 2, n = ",str(n_,0,0),".\n")
//  Foreach(A, dictionary {.Macro: "m_l3"})
  Foreach(A, dictionary {.Macro: "m_l10"})
#end

#macro m_l1(i_,n_)
  #debug concat("-> level 1, n = ",str(n_,0,0),".\n")
  Foreach(A, dictionary {.Macro: "m_l2"})
#end

Foreach(A, dictionary {.Macro: "m_l1"})


jr@swift:4:foreach$ povparse nesting.pov
  ...
==== [Parsing...] ==========================================================
-> level 1, n = 1.
--> level 2, n = 1.
----------> level 10, n = 1.
----------> level 10, n = 2.
----------> level 10, n = 3.
----------> level 10, n = 4.
File '/tmp/parse_fore.tmp' line 1: Parse Error: Expected 'object or directive',
 ) found instead
Fatal error in parser: Cannot parse input.
Render failed


jr@swift:5:foreach$ tail -n15 nesting.pov
#end

#macro m_l2(i_,n_)
  #debug concat("--> level 2, n = ",str(n_,0,0),".\n")
  Foreach(A, dictionary {.Macro: "m_l3"})
//  Foreach(A, dictionary {.Macro: "m_l10"})
#end

#macro m_l1(i_,n_)
  #debug concat("-> level 1, n = ",str(n_,0,0),".\n")
  Foreach(A, dictionary {.Macro: "m_l2"})
#end

Foreach(A, dictionary {.Macro: "m_l1"})


jr@swift:6:foreach$ povparse nesting.pov
  ...
==== [Parsing...] ==========================================================
-> level 1, n = 1.
--> level 2, n = 1.
---> level 3, n = 1.
---> level 3, n = 2.
  ...
---> level 3, n = 4.
--> level 2, n = 4.
---> level 3, n = 1.
---> level 3, n = 2.
---> level 3, n = 3.
---> level 3, n = 4.
File 'nesting.pov' line 327: Parse Warning: No objects in scene.

(the last with foreach in 'm_l3' commented out)

Post a reply to this message

hi WFP + TOK

"jr" <cre### [at] gmailcom> wrote:
> ...

rename 'm_l10' to, say, 'mFoo', and it just works.  well, the parser's still "at
it".  :-)  something to do with the actual name!


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> hi WFP + TOK
> ...
> rename 'm_l10' to, say, 'mFoo', and it just works.  well, the parser's still "at
> it".  :-)  something to do with the actual name!

was rushed, earlier.  just did another quick test just with names.  I thought
there might be a limit on the number of identifiers with some identical prefix,
but it isn't so straightforward; may try other stuff nearer weekend (if still
"aktuell").


regards, jr.


Script started on Tue 27 Apr 2021 14:46:10 BST

jr@swift:8:foreach$ c### [at] namebugpov
#version 3.8;

global_settings {assumed_gamma 1}

#include "foreach_d.inc"

#declare A = array [1] {1};

#macro mac_01(i_,n_)
  #debug concat("macro 01",".\n")
#end
#macro mac_02(i_,n_)
  #debug concat("macro 02",".\n")
#end
#macro mac_03(i_,n_)
  #debug concat("macro 03",".\n")
#end
#macro mac_04(i_,n_)
  #debug concat("macro 04",".\n")
#end
#macro mac_05(i_,n_)
  #debug concat("macro 05",".\n")
#end
#macro mac_06(i_,n_)
  #debug concat("macro 06",".\n")
#end
#macro mac_07(i_,n_)
  #debug concat("macro 07",".\n")
#end
#macro mac_08(i_,n_)
  #debug concat("macro 08",".\n")
#end
#macro mac_09(i_,n_)
  #debug concat("macro 09",".\n")
#end
#macro mac_10(i_,n_)
  #debug concat("macro 10",".\n")
#end
#macro mac_11(i_,n_)
  #debug concat("macro 11",".\n")
#end
#macro mac_12(i_,n_)
  #debug concat("macro 12",".\n")
#end

#for (i_, 1, 12)
  Foreach(A, dictionary {.Macro: concat("mac_",str(i_,-2,0))})
#end

jr@swift:9:foreach$ povparse !$
  ...
==== [Parsing...] ==========================================================
macro 01.
macro 02.
macro 03.
macro 04.
macro 05.
macro 06.
macro 07.
macro 08.
macro 09.
macro 10.
macro 11.
macro 12.
File 'namebug.pov' line 48: Parse Warning: No objects in scene.

Post a reply to this message

On 4/27/21 6:52 AM, jr wrote:
> hi,
> 

Played a few minutes with this just now and you can get your original 
code to run if you change the tail m_l10 macro to:

#macro m_l10(i_,n_)
   #debug concat("----------> level 10, n = ",str(n_,0,0),".\n")
   #fopen FID "parse_fore.tmp" write
   #write (FID,"#---\n")
   #fclose FID
#end

Though by the time it finishes it's consumed 1.6G or more of memory 
after maybe five minutes. My wild guess is all those Parse_String(s) end 
up sitting in memory until parsing is finished. In other words I think 
at the end we have unrolled more or less the whole recursion.

Parse_String is involved in what's happening, but it's not clear to me, 
why - some scope issue maybe as it comes after the last "call" to m_l10.

Bill P.

Post a reply to this message

On 4/27/21 7:56 PM, William F Pokorny wrote:
> On 4/27/21 6:52 AM, jr wrote:
>> hi,
>>
> 
> Played a few minutes with this just now and you can get your original 
> code to run if you change the tail m_l10 macro to:
> 
> #macro m_l10(i_,n_)




> #end
> 
...
> 
> Parse_String is involved in what's happening, but it's not clear to me, 
> why - some scope issue maybe as it comes after the last "call" to m_l10.
> 

OK.

Not 100% sure, in that I've not proven it by making all the 
'parse_fore.tmp' names unique, but I believe we are reading (or I 
suspect continuing to read) via a now back in scope #include to a common 
file with updated, and longer contents written in the previous deeper 
scope(1).

If in your include we add two lines to 'empty' the included file after 
including / dynamically interpreting the intended string:

#macro fore_exec(s_)
   #local fn_ = "parse_fore.tmp";
   #fopen fp_ fn_ write
   #write (fp_,s_)
   #fclose fp_
   #include fn_
   #fopen fp_ fn_ write  // Making file empty here fixes issue too
   #fclose fp_
#end

The code runs OK.

You ask, "So, this is an issue to which the strings.inc's
Parse_String() macro has always been exposed too?" Yes, I suspect so 
though I've not gone back to older versions of POV-Ray to try and 
reproduce the issue.

Is it worth further chasing and changing, if this is how v3.8 works 
today? I vote NO.

I'm thinking the practical fix is our 'Parse_String()'s should always 
empty the file just included.

We should remember too the Parse_String() method is a hack; It's bad 
practice to be writing and reading the same file name from multiple 
scene 'scopes' or invocations of POV-Ray.

(1) - POV-Ray re-reads files by file positions while working with macros 
and includes. What I think you hit on in changing the tail macro name 
and finding the code working, was changing to a macro name with the same 
(or shorter) length as (all) the earlier ones in the recursion. Meaning 
the file position markers didn't change (with respect to the file 
contents) in a way causing additional characters to be read from the 
'common' parse_fore.tmp file name. This theory can be tested by, say, 
changing the m_l10 name to say ml10 - should run?

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> Played a few minutes with this just now and you can get your original
> code to run if you change the tail m_l10 macro to:
>
> #macro m_l10(i_,n_)
>    #debug concat("----------> level 10, n = ",str(n_,0,0),".\n")
>    #fopen FID "parse_fore.tmp" write
>    #write (FID,"#---\n")
>    #fclose FID
> #end
>
> Though by the time it finishes it's consumed 1.6G or more of memory
> after maybe five minutes. My wild guess is all those Parse_String(s) end
> up sitting in memory until parsing is finished. In other words I think
> at the end we have unrolled more or less the whole recursion.
>
> Parse_String is involved in what's happening, but it's not clear to me,
> why - some scope issue maybe as it comes after the last "call" to m_l10.

all v strange.  my hunch is that Parse_String is involved indirectly only.  the
exact naming seems to matter, so I wonder what happens when 'm_l10' gets
inserted in the symbol table (cf TOK's error message).  the level of nesting --
thinking of calling it "voluntary recursion" :-) -- is not the problem I think,
attached a scene that nests/recurses to 19 levels, no problems (except time +
memory consumption ;-))  will try and find a little time in next days to play
with different naming "schemes".


regards, jr.

Post a reply to this message

Attachments:
Download 'n19tst.pov.txt' (3 KB)

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 4/27/21 7:56 PM, William F Pokorny wrote:
> > On 4/27/21 6:52 AM, jr wrote:
> >> ...
> Is it worth further chasing and changing, if this is how v3.8 works
> today? I vote NO.

(sorry, forgot)

agree it's no real problem, will make note in foreach docs that a handful of
levels (ie sane usage) is ok, beyond you take chances.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> ...
> all v strange.  my hunch is that Parse_String is involved indirectly only.  the
> exact naming seems to matter, ...

took same code as the original 'nesting.pov', changed all macro names to _not_
include numeric digits, and - fine.  hth.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> posting #4 of 'Foreach()', probably final.

posting a new version, #5.

some of the code has been refactored and tidied up.  many changes were
"cosmetic", there are now no comments within any macro, for instance, and the
layout of the '.Verbose' output has been adjusted.  all this has resulted in a
small, but measurable, improvement in "performance"; the 'n19tst.pov' posted
earlier in this thread went from ~4m7s to ~3m59s, and a "monkeyed" version, with
all comments and blank lines etc removed, saves another 6 seconds or so.

the optional '.Arg' has been .. de-coupled from its dependence on the '.Extra'
flag, providing additional flexibility from here on.

the html "man" page has been updated to reflect all changes, and another
(complete) demo scene has been added.  the 'fore_demo.pov' scene is now
accompanied by the (long) overdue 'fore_demo.ini' file.

feedback on how to make 'Foreach()' better will be appreciated.


enjoy, jr.

Post a reply to this message

Attachments:
Download 'foreach5.zip' (14 KB)