On 2/8/22 09:25, jr wrote:
> when will/can you make this available?

I'm aiming for March, but these fork information hooks are not yet even 
checked in. It's all the first pass at something I think workable and I 
fully expect to further polish the include file header and clean up the 
modified code.

That said, I've begun doing some sanity checking that performance 
related changes implemented over the past few months, hold for the -o3 
-ffast-math -march=native and configure --enable-lto compiles where I 
intend to use my povr branch personally.

The --enable-lto option, especially, is too painful to use during day to 
day development, but I certainly want to use it when really rendering 
scenes.

I've done a fair bit to slow povr code down too in fixing bugs and 
adding options. Trying too, to check none of those cause too much slow 
down as compiled and linked for real use.

Bill P.

Post a reply to this message

On 2/9/22 13:16, William F Pokorny wrote:
> On 2/8/22 09:25, jr wrote:
>> when will/can you make this available?
> 
> I'm aiming for March, but these fork information hooks are not yet even 
> checked in. It's all the first pass at something I think workable and I 
> fully expect to further polish the include file header and clean up the 
> modified code.
> 

Attached is a refinement of the forkversionhook.inc file. The more I 
thought about Bald Eagle's idea of versioning, the more I came to 
believe his "do versioning now" idea a good one. Previously, I basically 
had a release/tarball count, but that could easily change.

So, this version adopts and defines POV-Ray's Major, Minor, Revision, 
Patch set up as creates a new povr release/tarball counter for it's 
simpler 'version' number. Code wise, I also moved the value definitions 
into version.h alongside POV-Ray's normal versioning. That version.h 
file already also defines well what the values mean.

For the code check in I'm about to do there will be no difference 
between povr's minor version and the tarball count (next is 6). All the
other values are currently 0. Of course the earlier five povr tarballs 
have none of this fork hook code.

Bill P.

Post a reply to this message

Attachments:
Download 'forkversionhook.inc.txt' (5 KB)

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> Feedback? Thoughts?

only a couple of "things".  (sorry took a few days)

I (still) think that the .inc would be a good place to store other
'povr'-specific stuff too, like the ids.

from the looks of 'forkversionhook.inc' it will be only for 'povr', I'd have
hoped for a 'version.inc' which would work regardless, for any version and/or
fork of POV-Ray.

lastly, the constant + macro names are, um, a little verbose?  :-)

thanks.


regards, jr.

Post a reply to this message

On 2/14/22 10:57, jr wrote:
> hi,
> 
> William F Pokorny<ano### [at] anonymousorg>  wrote:
>> Feedback? Thoughts?
> only a couple of "things".  (sorry took a few days)

No worries.

> 
> I (still) think that the .inc would be a good place to store other
> 'povr'-specific stuff too, like the ids.
> 

Here, I wonder if perhaps the scale of change in the povr branch is 
being seen as less extensive than it in fact is. The new setidtypes.inc 
is one of several completely new include files.

Nearly all the remaining, shipped include files are substantially 
different. Only rand.inc hasn't been changed much - mostly because I see 
a good many puzzles there in code that I don't think many use. I'm not 
sure what to do povr's current rand.inc as yet.

The povr branch only ships 14 includes - to be 15 with forkversionhook.inc.

Lastly, an aim is that the 'forkversionhook.inc' file be extensible. 
That, in approach, its able to accommodate multiple forks / patches / 
branches. In other words, in just supporting multiple forks over time, 
it could grow substantially in size. I see no reason this file cannot be 
shared - at least to some degree - across forks.

> from the looks of 'forkversionhook.inc' it will be only for 'povr', I'd have
> hoped for a 'version.inc' which would work regardless, for any version and/or
> fork of POV-Ray.

I'll try to reduce / rework the mentions of povr.

The current forkversionhook.inc does work for any POV-Ray version - at 
least when I've tested it.

It's only povr specific when the SDL variable Dump_povr_fork_ is set on 
inclusion. Then it prints documentation as to what povr fork_*() 
constants are available and stops - and it only does that much if the 
fork being run is in fact 'povr'. I'll add a commented #debug where I 
had an empty else block indicating something which could be printed when
the fork hook is not found. It's commented because I don't want to 
default to always printing something on inclusion of
forkversionhook.inc. Regular include use will be to make the 
'Have_f_odd_fork_hook()' macro available to scene code and nothing more.

> Have_f_odd_fork_hook()
> lastly, the constant + macro names are, um, a little verbose?:-)
> 

Agree.

In general, I think I'll reduce the f_odd() visibility. No reason day to 
day users need to know this the internal 'hook'.

I also think you're right it's better to call the fork_*()s, constants, 
rather than functions.

Beyond that I'll take you to mean, perhaps, something like the following:

forkversionhook.inc -->  forkhooks.inc ?

Have_f_odd_fork_hook() --> HaveForkConstants() ?

Dump_povr_fork_  --> Fork_povr ?

...I'm less a fan of shortening the constant pairs, but...

fork_str(n) --> frkstr(n) ?
fork_val(n) --> frkval(n) ?

Better?

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 2/14/22 10:57, jr wrote:
> > ...
> > I (still) think that the .inc would be a good place to store other
> > 'povr'-specific stuff too, like the ids.
> >
>
> Here, I wonder if perhaps the scale of change in the povr branch is
> being seen as less extensive than it in fact is. ...
> In other words, in just supporting multiple forks over time,
> it could grow substantially in size. I see no reason this file cannot be
> shared - at least to some degree - across forks.

yes there is much divergence, and I had not thought "forward", ie after a number
of subsequent releases, so scratch the thought.  (I focus perhaps too much on
"everything in one place", anyway)


> > from the looks of 'forkversionhook.inc' it will be only for 'povr', I'd have
> > hoped for a 'version.inc' which would work regardless, for any version and/or
> > fork of POV-Ray.
>
> I'll try to reduce / rework the mentions of povr.
>
> The current forkversionhook.inc does work for any POV-Ray version - at
> least when I've tested it.  ...
> I'll add a commented #debug where I
> had an empty else block indicating something which could be printed when
> the fork hook is not found. It's commented because I don't want to
> default to always printing something on inclusion of
> forkversionhook.inc. Regular include use will be to make the
> 'Have_f_odd_fork_hook()' macro available to scene code and nothing more.

works as in parses, but (empty else branch) no feedback when it isn't povr.  I
think that I'd hoped for a situation where POV-Ray proper and (eventually) all
forks supply a version.inc which tells, basically, whether or not "official";
forks, and POV-Ray, are then free to add constants and or macros to probe
further.


> > Have_f_odd_fork_hook()
> > lastly, the constant + macro names are, um, a little verbose?:-)
> ...
> Beyond that I'll take you to mean, perhaps, something like the following:
>
> forkversionhook.inc -->  forkhooks.inc ?
>
> Have_f_odd_fork_hook() --> HaveForkConstants() ?

:-)  'forkversion.inc', actually.  it's the "hook" that's .. surplus in my view.
 I don't mind the underscores, think they aid readability.


> ...I'm less a fan of shortening the constant pairs, but...
>
> fork_str(n) --> frkstr(n) ?
> fork_val(n) --> frkval(n) ?
>
> Better?

ouch!  personally (if it mattered) 'fork_{str,val}' is fine, though I'd probably
write 'fork_{key,val}'.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:

just to clarify.

> ...
> think that I'd hoped for a situation where POV-Ray proper and (eventually) all
> forks supply a version.inc which tells, basically, whether or not "official";

I don't mean official as in compiled-by.  "official" returned from a version.inc
would tell me that the scene's SDL will work as detailed in the official
wiki/docs.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:

> Lastly, an aim is that the 'forkversionhook.inc' file be extensible.
> That, in approach, its able to accommodate multiple forks / patches /
> branches. In other words, in just supporting multiple forks over time,
> it could grow substantially in size. I see no reason this file cannot be
> shared - at least to some degree - across forks.

It's just a text file.  And it's for POV-Ray - so just how many edits and forks
there will be ....   I mean, even if it was War & Peace in ASCII, it still
wouldn't be that large of a file.
A functional option, one I'm not sure I like ATM, is to have a version
_directory_ with a tree of inc files that the main file includes and executes -
only if they exist.   Then the individual users can grow or prune the contents
of the directory as they see fit.  Still think that for any given fork, a
generic named file should call the most recent name-with-version.inc
That way people can search the forums (fora?) and easily see if they have the
proper/latest version of what they're looking for.   Even if it's just File1,
File2, File3 ....   it at least makes the filename different, and gives an
easily identifiable sequential progression.

> In general, I think I'll reduce the f_odd() visibility. No reason day to
> day users need to know this the internal 'hook'.

You can have silent and verbose modes.
Silent is for everyday use, verbose can be for anyone looking to figure
something out.   Just set a flag, look for the presence of a dummy file...


As for constant, function, and macro names, there's a school of thought that if
your names are descriptive enough, the code is "self documenting" and doesn't
require comments.   I think this works well for code that isn't long on the
first few (re)write(s) and only gets edited every so often.   Using long names
in day to day use is a DRAG.

Post a reply to this message

On 2/15/22 02:11, jr wrote:
> "jr" <cre### [at] gmailcom> wrote:
> 
> just to clarify.
> 
>> ...
>> think that I'd hoped for a situation where POV-Ray proper and (eventually) all
>> forks supply a version.inc which tells, basically, whether or not "official";
> 
> I don't mean official as in compiled-by.  "official" returned from a version.inc
> would tell me that the scene's SDL will work as detailed in the official
> wiki/docs.
> 

Thinking.

Nobody can truly declare particular behavior without being the one 
providing a pre-compiled POV-Ray package. Further, the compiled program 
needs to be linked with, or use, fixed library dependencies. The 
official status is indicated by the POV-Team only for windows binaries 
where the current libraries are part of the git tracked code base.

Those binaries only state official status in the text output while 
setting up the parser to die on the 'unofficial' keyword. There isn't a 
way to query official / unofficial status from within the SDL.

We want to be able to set up scenes / includes with conditionals so we 
can run compiles from nominally unmodified source alongside forks like 
povr. This rules out the use of the 'unofficial' keyword for starters.

We also want something like 'unofficial' which dies for non-windows 
POV-Team provided binaries where we know the code will only run with a 
given fork(2).

I didn't call 'forkversion.inc' 'version.inc' to allow for an official 
POV-Team shipped 'version.inc'.

I suppose OK for forks to provide 'version.inc' files ahead of it 
potentially existing in POV-Team releases. Attached two povr include 
files in 'version.inc'(1) and 'forkversion.inc'. The latter with 
additional updates - shorter names, rewording, etc.

Comments welcome.

Aside: I'm unsure when / if to change the povr code to force all 
incoming SDL to run strictly at v3.8+. Internally, I've slowly been 
tossing code for backward compatibility into the bit bucket. V3.8+ SDL 
is all I intend to support. However, as many are still working with 
older releases, scenes and includes, I've been maintaining the parser in 
a state that allows earlier version settings. It's useful when looking 
at posted issues...

Bill P.

(1) - Uses file_exist for 'forkversion.inc' / 'version.inc', which can 
be problematic in the way file_exist with includes can be. However, I 
see no better method for a backward compatible 'version.inc'.

(2) - Thinking with povr I'll now use both 'unofficial' and code like:

#if (file_exists("version.inc"))
     #include "version.inc"
#end
#if (!Fork_povr)   // This POV-Ray SDL code requires the povr fork.
     #error "This POV-Ray SDL code requires the povr fork."
#end

Post a reply to this message

Attachments:
Download 'version.inc.txt' (4 KB) Download 'forkversion.inc.txt' (5 KB)

On 2/15/22 07:07, Bald Eagle wrote:
> It's just a text file.

True. It's also an include which the parser must digest. With animations 
the parser eats all includes, again, for each frame. Though users often 
include .inc files by the handful(1), being more selective about what 
needs to be included is more efficient.

I'll let your other ideas and thoughts bang around in my head for a while.

Bill P.

(1) - Via stdinc.inc as often as not.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> ...
> There isn't a
> way to query official / unofficial status from within the SDL.

yet.


> We want to be able to set up scenes / includes with conditionals so we
> can run compiles from nominally unmodified source alongside forks like
> povr. This rules out the use of the 'unofficial' keyword for starters.
>
> We also want something like 'unofficial' which dies for non-windows
> POV-Team provided binaries where we know the code will only run with a
> given fork(2).
>
> I didn't call 'forkversion.inc' 'version.inc' to allow for an official
> POV-Team shipped 'version.inc'.

chain-loading via 'file_exists()' is nice + "clean" (imo).


> ...
> Comments welcome.
>
> Aside: I'm unsure when / if to change the povr code to force all
> incoming SDL to run strictly at v3.8+. Internally, I've slowly been
> tossing code for backward compatibility into the bit bucket. V3.8+ SDL
> is all I intend to support. However, as many are still working with
> older releases, scenes and includes, I've been maintaining the parser in
> a state that allows earlier version settings. It's useful when looking
> at posted issues...

it would be interesting to see proposals for an updated v4+ SDL.

out of interest, how difficult would it be to have two separate parsers?  one
for pre 3.7/8 compatibility, one for newer style.  (I'm guessing that the
executable would not be all that much larger, and separation might bring other
benefits)


have only spent a few minutes looking over the attached, seem ok (bar copy/paste
typos 'str_var' in 'forkversion.inc'.)


regards, jr.

Post a reply to this message

On 2/17/22 02:39, jr wrote:
> chain-loading via 'file_exists()' is nice + "clean" (imo).
> 

I'm not completely sure I'm following what you mean by chain-loading?

I've adopted the use of file_exists() on deciding to ship a version.inc 
with povr. The overall implementation is done in a 
scene->version.inc->forkversion.inc way. Is this what you mean by 
chain-loading?

---
file_exist() is one of those things which is clean until it's not. :-)

Think about how we've often shipped binaries intended to run on a 
earlier install base. There is build infrastructure to support this 
model of update. A model somewhat workable because official POV-Ray 
supports backward compatibility. It's problematic in that to safely 
align with any given binary you need the binary's includes too - 
something that in POV-Ray proper requires a full install.

Think too about how the search path is user modifiable.

Anyhow, it's the method we have. I agree too for normal users with a 
single installation of POV-Ray, and relatively constant include files, 
file_exist is safe enough.

> 
>> ...
>> Comments welcome.
>>
>> Aside: I'm unsure when / if to change the povr code to force all
>> incoming SDL to run strictly at v3.8+. Internally, I've slowly been
>> tossing code for backward compatibility into the bit bucket. V3.8+ SDL
>> is all I intend to support. However, as many are still working with
>> older releases, scenes and includes, I've been maintaining the parser in
>> a state that allows earlier version settings. It's useful when looking
>> at posted issues...
> it would be interesting to see proposals for an updated v4+ SDL.

Practically, I don't think I've got the bandwidth for anything 
revolutionary in the povr fork.

It would probably be TCL if I do get around to trying something radical. 
I have a lot of that going today - as you know - but the implementation 
writes SDL. In other words, it still leans on the current SDL parsers.

> 
> out of interest, how difficult would it be to have two separate parsers?  one
> for pre 3.7/8 compatibility, one for newer style.  (I'm guessing that the
> executable would not be all that much larger, and separation might bring other
> benefits)
> 

My up front answer - difficult. The parser is still too integrated with 
the run time code. As something part of my povr branch it's not the most 
unattractive idea because it tangles me in two different parser 
implementations.

Aside: I've thought about hacking versions of POV-Ray in a way where 
they could be run as 'parse to some intermediate form and stop' 
utilities. And that thought reminds me I want to implement an actual 
#stop after parse directive. I use #error, but it's an ugly way to stop 
after the parsing stage.

Anyway, thanks for the ideas! As with the ini and options setting 
approaches, I'm still uncertain where the povr code will go.

> 
> have only spent a few minutes looking over the attached, seem ok (bar copy/paste
> typos 'str_var' in 'forkversion.inc'.)

Thanks. I believe that fixed.

And apologies. I must not of finished the editing of forkversion.inc (or 
somehow didn't do a last save...) I picked up a number of other mistakes 
'I thought' I'd fixed.

Bill P.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 2/17/22 02:39, jr wrote:
> > chain-loading via 'file_exists()' is nice + "clean" (imo).
> >
>
> I'm not completely sure I'm following what you mean by chain-loading?

do file_exists (filename)
if yes, run/include that file.
In THAT file is _another_ file_exists (filename)... and so on

daisy-chain everything together until you hit a dead-end.



> Aside: I've thought about hacking versions of POV-Ray in a way where
> they could be run as 'parse to some intermediate form and stop'
> utilities. And that thought reminds me I want to implement an actual
> #stop after parse directive. I use #error, but it's an ugly way to stop
> after the parsing stage.

Yes.
More useful would be to have that as a "function" that returns some sort of
state variable, visible to the OS/shell

I've also thought about a mechanism to "stop" and then _reparse_ the SDL file,
such that certain things could be scanned, written to file, or otherwise do a
pre-parse-and-render setup.  Maybe something like: #reparse 3
POV-Ray would parse until it hit that, decrement a counter, etc, until it read
through that directive at counter = 0.

I suppose we have workarounds for stuff like this.

I've also wondered about write-protecting files in some way to make editing them
a conscious and fully purposeful action.   Like locking the cells in a
spreadsheet so that critical, complicated formulas don't get edited by someone
trying to "improve" things.  ;)

Silly idea:   certain image formats have header sections where you can embed
virtually anything.   Would it be possible to have POV-Ray store scene files in
an image file format, with the SDL embedded in the header?  Not sure where I was
going with this a second ago - but just thinking about such a thing might spark
some new ideas about structuring scene, includes, etc.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 2/17/22 02:39, jr wrote:
> > chain-loading via 'file_exists()' is nice + "clean" (imo).
> >
>
> I'm not completely sure I'm following what you mean by chain-loading?

as BE says.  what I tried to say is something like the mechanism is nice/clean
because it all is/can be transparent to the user (as your code snippet
demonstrates).


> > > Thinking
> ...
> It would probably be TCL if I do get around to trying something radical.

please do!  ;-)


> I have a lot of that going today - as you know - but the implementation
> writes SDL. In other words, it still leans on the current SDL parsers.

I guess that a C(++) extension for Tcl, to drive the VM from Tcl, would do it
:-), access through some "specialising" packages/modules perhaps.


> > out of interest, how difficult would it be to have two separate parsers?  one
> > for pre 3.7/8 compatibility, one for newer style.  (I'm guessing that the
> > executable would not be all that much larger, and separation might bring other
> > benefits)
>
> My up front answer - difficult. The parser is still too integrated with
> the run time code. As something part of my povr branch it's not the most
> unattractive idea because it tangles me in two different parser
> implementations.

ah, naively I'd thought that the existing CLI/ini 'version' could be (ab)used,
ie if not given start with forked parser, if any file read needs <3.8 stop/error
tell user either needs option switch (to start with the "compatible") or edit
offending file.


> Aside: I've thought about hacking versions of POV-Ray in a way where
> they could be run as 'parse to some intermediate form and stop'
> utilities. And that thought reminds me I want to implement an actual
> #stop after parse directive. I use #error, but it's an ugly way to stop
> after the parsing stage.

agree with BE, being able to return a status from a scene would be neat,
particularly if it's available for 'post_frame_command's.

also re his "silly idea", SDL (and or pre-parsed VM calls) storage -- how about
DB (sqlite) storage?


> Anyway, thanks for the ideas! As with the ini and options setting
> approaches, I'm still uncertain where the povr code will go.

:-)  you'll have to tell me when to stop.


regards, jr.

Post a reply to this message

hi,

"Bald Eagle" <cre### [at] netscapenet> wrote:
> ...
> Silly idea:   certain image formats have header sections where you can embed
> virtually anything.   Would it be possible to have POV-Ray store scene files in
> an image file format, with the SDL embedded in the header?  Not sure where I was
> going with this a second ago - but just thinking about such a thing might spark
> some new ideas about structuring scene, includes, etc.

look at attached with:

  $ hexdump -C hu_8.png | less

that what you mean?  ;-)

anyway, long story short, you can do in Tcl, for .png files.  install the
'Tcllib' package, then check out the man page (section n on my box) for 'png'.


regards, jr.

Post a reply to this message

Attachments:
Download 'hu_8.png' (17 KB)

Preview of image 'hu_8.png'

"jr" <cre### [at] gmailcom> wrote:
> hi,
>
> "Bald Eagle" <cre### [at] netscapenet> wrote:
> > ...
> > Silly idea:   certain image formats have header sections where you can embed
> > virtually anything.   Would it be possible to have POV-Ray store scene files in
> > an image file format, with the SDL embedded in the header?  Not sure where I was
> > going with this a second ago - but just thinking about such a thing might spark
> > some new ideas about structuring scene, includes, etc.
>
> look at attached with:
>
>   $ hexdump -C hu_8.png | less

Well, not exactly.   I think I wrote that on my way out to work, so it was just
a short brain-dump.

What I was actually thinking was that POV-Ray could use an image format as the
actual file format for storing the SDL text, and the editor could extract it
from that, display it, allow for editing, and re-save into the image file
wrapper.

Why do this:
I guess the idea was that since POV-Ray already has the ability to
parse/interpret image file formats already, it might be possible to somehow
embed a series of text files into the image header - like includes, etc and use
it kind of like a zip file.

Also, then we could have the image and the code that generated it all in one
place, so they aren't separated and lost/confused.

It might also be possible to implement security by obscurity, to protect certain
parts of code that an author might not want users casually messing with.  If you
really wanted to edit that code, you'd have to work for it, rather than just
"Ooops!  I broke it!"

But binary copy, hexdump, Tcp, and other tools might provide a sort of
functional work-around / proof of concept to see how a suite of files could
potentially be used ....

Post a reply to this message

On 2/28/22 06:47, jr wrote:
> look at attached with:
> 
>    $ hexdump -C hu_8.png | less
> 
> that what you mean?;-)
> 
> anyway, long story short, you can do in Tcl, for .png files.  install the
> 'Tcllib' package, then check out the man page (section n on my box) for 'png'.

Neat.

What particular commands and versions of TCL and Tcllib are you using?

It's been maybe 4-5 years since I played in the tcl/png sandbox but, not 
too long after tcl 8.6 was first released, I found the png support from 
within tcl/tk/tcllib buggy. Supposing you are finding it better?

I was primarily trying to load/create/modify/save actual image channel 
data but, I don't recall even the particular 'TCL' packages and commands 
I tried at the moment...

Bill P.

Post a reply to this message

hi,

"Bald Eagle" <cre### [at] netscapenet> wrote:
> ...
> What I was actually thinking was that POV-Ray could use an image format as the
> actual file format for storing the SDL text, and the editor could extract it
> from that, display it, allow for editing, and re-save into the image file
> wrapper.
>
> Why do this:
> I guess the idea was that since POV-Ray already has the ability to
> parse/interpret image file formats already, it might be possible to somehow
> embed a series of text files into the image header - like includes, etc and use
> it kind of like a zip file.
>
> Also, then we could have the image and the code that generated it all in one
> place, so they aren't separated and lost/confused.

- keeping "image and the code" in one place, along with the includes, bg
image(s) etc, would be ideal.  a project oriented approach.  and zip would be a
good format.

- hate the idea/concept of "allow for editing, and re-save into the image file
wrapper".  if anything the other way round, keep everything plain human-readable
text and include image(s) and other binary content 'base64' encoded.

anyway, thanks, have a clearer picture now of the "silly idea".


> It might also be possible to implement security by obscurity, to protect certain
> parts of code that an author might not want users casually messing with.  If you
> really wanted to edit that code, you'd have to work for it, rather than just
> "Ooops!  I broke it!"

also not keen on this.  if an author feels .. so strongly, well, then don't
publish.  simples.


> But binary copy, hexdump, Tcp, and other tools might provide a sort of
> functional work-around / proof of concept to see how a suite of files could
> potentially be used ....

why binary anyway?  what's wrong with a plain text (or utf8) encoding?


regards, jr.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> ...
> What particular commands and versions of TCL and Tcllib are you using?

(typed)

% set tcl_version
8.6
% package require crc32
% package require png
% png::addComment foo.png theTag theValue

and using a 'tcllib1.18' (though mean to upgrade to 1.19).


> It's been maybe 4-5 years since I played in the tcl/png sandbox but, not
> too long after tcl 8.6 was first released, I found the png support from
> within tcl/tk/tcllib buggy. Supposing you are finding it better?

guess so, cannot recall any problem(s) with 'image' and 'photo' etc.


> I was primarily trying to load/create/modify/save actual image channel
> data but, I don't recall even the particular 'TCL' packages and commands
> I tried at the moment...

fwiw 'df3util' can split PNGs into rgba (and stored in a DF3, but there's the
'export' command).

hope to post a 'compare(1)' image in the next days, showing some/much difference
for the sphere_sweep between 'povr' and stock POV-Ray.  will need to
systematically try with 'sturm' and 'tolerance' etc first though.


regards, jr.

Post a reply to this message

On 28/02/2022 12:47, jr wrote:
> hi,
> 
> "Bald Eagle" <cre### [at] netscapenet> wrote:
>> ...
>> Silly idea:   certain image formats have header sections where you can embed
>> virtually anything.   Would it be possible to have POV-Ray store scene files in
>> an image file format, with the SDL embedded in the header?  Not sure where I was
>> going with this a second ago - but just thinking about such a thing might spark
>> some new ideas about structuring scene, includes, etc.
> 
> look at attached with:
> 
>   $ hexdump -C hu_8.png | less
> 
> that what you mean?  ;-)


Really funny !
<https://en.wikipedia.org/wiki/Steganography>


-- 
Kurtz le pirate
Compagnie de la Banquise

Post a reply to this message

thinking some more..

William F Pokorny <ano### [at] anonymousorg> wrote:
> Neat.

and super-easy, _and_ since Tcl also runs on Redmond's finest, potentially of
interest to every user of POV-Ray who wants to publish/distribute an image.
therefore, imo, in a separate thread, we ("all interested parties" :-)) should
have a discussion to come up with an informal set of useful "recommended"
tags/keys. think will start one tomorrow (unless you/someone gets there first
</grin>).


@kurtz

"covered writing" indeed, as many (most?) image viewers simply ignore the info.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:

> - keeping "image and the code" in one place, along with the includes, bg
> image(s) etc, would be ideal.  a project oriented approach.  and zip would be a
> good format.
>
> - hate the idea/concept of "allow for editing, and re-save into the image file
> wrapper".  if anything the other way round, keep everything plain human-readable
> text and include image(s) and other binary content 'base64' encoded.
>
> anyway, thanks, have a clearer picture now of the "silly idea".

I'm not sure I care too much exactly how it gets implemented.  I was under the
impression that opening such an image file with a hex editor would result in the
text part looking like --- plain text.  It's not like plain text is "special" -
it all has to get displayed on the screen by something that interprets the ASCII
values anyway.   At least that's the way it works in my head.

What about:
http://textbundle.org/
https://www.macsparky.com/blog/2014/11/the-textbundle-format/

I posted about this idea a while back - there was some other file format that
sort of acted like concatenated text files....   maybe it was a 'nix format...
I can't remember.


> > It might also be possible to implement security by obscurity, to protect certain
> > parts of code that an author might not want users casually messing with.  If you
> > really wanted to edit that code, you'd have to work for it, rather than just
> > "Ooops!  I broke it!"
>
> also not keen on this.  if an author feels .. so strongly, well, then don't
> publish.  simples.

Right, but the idea is just to make the code available and usable, but not right
there in the editor where it can easily be inadvertently corrupted.  I often
write spreadsheets with "locked" cells - but the password is sitting right there
in a documentation tab in case someone wants to edit it in the future.   But in
daily use, they can't get messed up by accident, which is easy to do.

I would like a project-oriented format/editor.  Open the project, and the main
file is in the first tab (Maybe the second, if there's a documentation tab) and
the includes and image files are available in other tabs.
OpenOffice allows use of hyperlinks in the spreadsheet, which I've used to
navigate around a sheet or between tabs.  It would be cool to specify image
filenames, and be able to click on those and get a preview, then "go back" and
edit the SDL.  Working on some code and need to find out what some value an
Lvalue has?   Click it and you jump back to the last #declare for that Lvalue.

But I think just starting off with the basic bundle idea would solve a lot of
organizational and archival issues.

> > But binary copy, hexdump, Tcp, and other tools might provide a sort of
> > functional work-around / proof of concept to see how a suite of files could
> > potentially be used ....
>
> why binary anyway?  what's wrong with a plain text (or utf8) encoding?

How do you keep the different text/SDL files separate, and maybe even package
images for mapping together?   Isn't a zip a binary format?

Post a reply to this message

Le 2022-03-01 à 12:06, kurtz le pirate a écrit :

> 
> Really funny !
> <https://en.wikipedia.org/wiki/Steganography>
> 
> 

We see some obvious example of that in MANY movies from the 1950's to 
1980's when someone use some aperture mask to reveal a hidden message in 
a letter.
I've seen at least one movie where we see a character actually writing 
such a message.

Post a reply to this message

hi,

"Bald Eagle" <cre### [at] netscapenet> wrote:
> "jr" <cre### [at] gmailcom> wrote:
>
> > - keeping "image and the code" in one place, along with the includes, bg
> > image(s) etc, would be ideal.  a project oriented approach.  and zip would be a
> > good format.
> >
> > - hate the idea/concept of "allow for editing, and re-save into the image file
> > wrapper".  if anything the other way round, keep everything plain human-readable
> > text and include image(s) and other binary content 'base64' encoded.
> >
> > anyway, thanks, have a clearer picture now of the "silly idea".
>
> I'm not sure I care too much exactly how it gets implemented.  I was under the
> impression that opening such an image file with a hex editor would result in the
> text part looking like --- plain text.  It's not like plain text is "special" -
> it all has to get displayed on the screen by something that interprets the ASCII
> values anyway.   At least that's the way it works in my head.

I suggested 'hexdump' because it is "safe", ie no stray control characters to
mess with yr terminal/console.  for comfortable viewing one could use
'png::getComments', or even simply:

  $ strings -n7 foo.png | less


> What about:
> http://textbundle.org/
> https://www.macsparky.com/blog/2014/11/the-textbundle-format/

see below.


> I posted about this idea a while back - there was some other file format that
> sort of acted like concatenated text files....   maybe it was a 'nix format...
> I can't remember.

unsure.  are you thinking of "literate programming"?  cf Knuth.


> > > It might also be possible to implement security by obscurity, to protect certain
> > > parts of code that an author might not want users casually messing with.  ...
> > also not keen on this.  if an author feels .. so strongly, well, then don't
> > publish.  simples.
>
> Right, but the idea is just to make the code available and usable, but not right
> there in the editor where it can easily be inadvertently corrupted.  I often
> write spreadsheets with "locked" cells - but the password is sitting right there
> in a documentation tab in case someone wants to edit it in the future.   But in
> daily use, they can't get messed up by accident, which is easy to do.

sure.  use a viewer like 'less' and only invoke the editor (by pressing 'v')
when intending to edit.  something like an .inc or other data one could make
read-only.


> I would like a project-oriented format/editor.  Open the project, and the main
> file is in the first tab (Maybe the second, if there's a documentation tab) and
> the includes and image files are available in other tabs.

though not using it at present, isn't that much like 'qtpovray's interface?


> OpenOffice allows use of hyperlinks in the spreadsheet, which I've used to
> navigate around a sheet or between tabs.  It would be cool to specify image
> filenames, and be able to click on those and get a preview, then "go back" and
> edit the SDL.  Working on some code and need to find out what some value an
> Lvalue has?   Click it and you jump back to the last #declare for that Lvalue.
>
> But I think just starting off with the basic bundle idea would solve a lot of
> organizational and archival issues.

yes.  think 'fossil'.  seriously.  as long as the project is a single directory
tree, that will address half of your needs.  (g)vim and a 'tags' file for
navigation will provide the other half.


> How do you keep the different text/SDL files separate, and maybe even package
> images for mapping together?   Isn't a zip a binary format?

and each fossil "repository" is a single file too, ie the same convenience
handling/backing up.


regards, jr.

Post a reply to this message

On 3/1/22 07:55, jr wrote:
> hi,
> 
> William F Pokorny <ano### [at] anonymousorg> wrote:
>> ...
>> What particular commands and versions of TCL and Tcllib are you using?
> 
> (typed)
> 
> % set tcl_version
> 8.6
> % package require crc32
> % package require png
> % png::addComment foo.png theTag theValue
> 
> and using a 'tcllib1.18' (though mean to upgrade to 1.19).
> 

Thanks.

Forgot TCL breaks the patch level out... What does the shell command:

echo 'puts [info patchlevel];exit 0' | tclsh

give you?

> 
>> It's been maybe 4-5 years since I played in the tcl/png sandbox but, not
>> too long after tcl 8.6 was first released, I found the png support from
>> within tcl/tk/tcllib buggy. Supposing you are finding it better?
> 
> guess so, cannot recall any problem(s) with 'image' and 'photo' etc.
> 
> 
>> I was primarily trying to load/create/modify/save actual image channel
>> data but, I don't recall even the particular 'TCL' packages and commands
>> I tried at the moment...
> 
> fwiw 'df3util' can split PNGs into rgba (and stored in a DF3, but there's the
> 'export' command).
> 

Thanks for the reminder. At the time I was aiming for a set of new 
commands in my TCL SDL wrapper.

Aside: I found out at some point the netpbm pam RGBA format is, less the 
short pam header and the addition of a short DF3 header, an exactly DF3 
equivalent file layer/plane. This means you can create / manipulate DF3s 
quickly with that tool set too.

> hope to post a 'compare(1)' image in the next days, showing some/much difference
> for the sphere_sweep between 'povr' and stock POV-Ray.  will need to
> systematically try with 'sturm' and 'tolerance' etc first though.
> 

I'm interested in the results you see from compare. I'm not a big 
imagemagick user.

--- sphere_sweep...

The povr sphere_sweep is substantially different in implementation being 
one of the shapes which leans on the generic inbuilt solvers. I did a 
large amount of work on those solvers some years ago.

There are at least a dozen numerical and control issues in that shape 
alone in POV-Ray proper. Much fixed or made much better in povr - but 
issues remain. Ultimately, I believe the code for sphere_sweeps needs to 
be completely re-written!

With sphere_sweep 'tolerance' my recollection is it does nothing in povr 
because in POV-Ray proper results are unstable due - another half dozen 
reasons.

First up, being the sphere_sweep is one our shapes where transforms get 
flattened (removed) if possible to improve performance. However, you 
cannot always flatten depending upon the particular transforms. This 
means the 'numerical meaning' of the 'tolerance' values (also things 
like no_bump_scale) are not themselves stable depending upon user scene 
transformations. Yep, this might be why the tolerance value was brought 
out as a control originally.

...And sturm...

In official POV-Ray releases, across many shapes, you are not always 
truly turning the sturm based solver on and off!

IIRC. The sphere_sweep sturm is always on internally (the polynomial 
orders are beyond 4) - excepting linear sweeps where it's hard coded 
always off (an unfortunate choice).

There's a general issue in official POV-Ray where, for auxiliary 
internal polynomials (ie inside tests), POV-Ray isn't changing to a 
sturm solver in a way aligned with the mainstream ray-surface solvers 
for some shapes - and sphere_sweep is one. Probably done for better 
performance too or, maybe believing the fixed solvers are 'always' the 
better choice. In any case, mixing solver approaches is not the best for 
'numerical alignment.'

In general! With povr, sturm off always uses a fixed solver where 
polynomial order allows it. Where sturm on, povr's version of the sturm 
solver is always used or, if the order is >4 'sturm on' will sometimes - 
as with sphere_sweeps(1) - mean use a 'better' sturm based solver.

(1) IIRC. It's currently doing order reduction after each root is found.

The takeaway is sturm behavior in sphere_sweeps is mostly fixed in 
official POV-Ray. In povr 'sturm on' always means "use a better sturm 
based solver."

Encyclopedia sets worth of detail when you get into the solvers... My 
head spins round and round! :-(

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/1/22 07:55, jr wrote:
> > ...
> > % set tcl_version
> > 8.6
> > ...
> Thanks.
>
> Forgot TCL breaks the patch level out... What does the shell command:
>
> echo 'puts [info patchlevel];exit 0' | tclsh
>
> give you?

8.6.5 (also have a 8.6.1 still).


> ...
> Aside: I found out at some point the netpbm pam RGBA format is, less the
> short pam header and the addition of a short DF3 header, an exactly DF3
> equivalent file layer/plane. This means you can create / manipulate DF3s
> quickly with that tool set too.

interesting, thanks.


> > hope to post a 'compare(1)' image in the next days, showing some/much difference
> > ...
> I'm interested in the results you see from compare. I'm not a big
> imagemagick user.

will need a couple of days or so.  read about sturm + sphere_sweep in
'revisions.txt', hence wanting to re-run + try things first.



@Cousin

the included-with-the-os Pinball was my .. vice, in the 3.11 heydays.  :-)


regards, jr.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> ...
> I'm interested in the results you see from compare. I'm not a big
> imagemagick user.

unarchiving the attached will create a (sub)directory 'wfp' (;-)) with a bunch
of files.  check the assumptions made (eg povray == stock POV-Ray) in the
'run-sh' file, then run that, ie 'sh run_sh', to render the images.  the compare
output will be in sub-dir 'c'.  cheers.


regards, jr.

Post a reply to this message

Attachments:
Download 'sweep.tar.xz.dat' (2 KB)

"jr" <cre### [at] gmailcom> wrote:

forgot to say, using alpha.9945627 and most recent povr.


> William F Pokorny <ano### [at] anonymousorg> wrote:
> > ...
> > I'm interested in the results you see from compare. I'm not a big
> > imagemagick user.

same here, rarely need/use them, except 'identify'.  woke early, so made this.
:-)


regards, jr.

Post a reply to this message

Attachments:
Download 'wfp_cmp.png' (69 KB)

Preview of image 'wfp_cmp.png'

On 3/5/22 02:49, jr wrote:
> "jr" <cre### [at] gmailcom> wrote:
> 
> forgot to say, using alpha.9945627 and most recent povr.
> 
> 
>> William F Pokorny <ano### [at] anonymousorg> wrote:
>>> ...
>>> I'm interested in the results you see from compare. I'm not a big
>>> imagemagick user.
> 
> same here, rarely need/use them, except 'identify'.  woke early, so made this.
> :-)
> 
> 
> regards, jr.

Somewhat busy with real life at the moment, but I did run your code 
yesterday evening and saw more or less what you just posted.

I won't be able to do more until later today at best, but the short of it:

- Where you set up sphere_sweeps with orthographic cameras (or rays say 
due parallel lights) is a worst case set up for sphere_sweeps because 
the initial ray-'virtual-surface' polynomial collapses to duplicate 
roots (at least it would always if we had infinite floating point accuracy).

- The above situation means you must use 'sturm' with the povr branch 
because povr's general solvers are all a lot more accurate(a) than the 
official POV-Ray solvers. :-) The detailed explanation is long...

- The color shift (the pink compare result) I'm pretty sure is due the 
povr fork restoring the sRGB block to png outputs. The sRGB block got 
dropped - due complicated reasons I forget at the moment - at some point 
during v3.71/v3.8 development. The sRGB png block is there in v3.7 png 
output and in recent povr(b) tarballs. My compares with netpbm - which 
default to bt709 gamma handling for both file outputs - show no color 
shift. There are other ways to test the png being missing being the 
reason for the pink 'compare(c,d)' result, but not done it and not sure 
I will.

Bill P.

(a) - The banding with linear sweeps (quadratic) comes from the povr 
default being the common quadratic solver which - like POV-Ray's version 
- has been hacked to return only one root on getting duplicate roots 
where it doesn't by numerical noise see or force two - and this happens 
in numerical bands with povr's more accurate quadratic solver. Where the 
less capable povr sturm solver used (povr's sturm off with >4 orders) we 
now get speckled results for similar, the solver is more accurate, 
reasons. (The duplicate roots are blinking in and out ray to ray)

(b) - Bring both your v3.8 and povr png files up in an editor and you'll 
see the text near the top 'sRGB' in povr (and v3.7) output. This test is 
missing in any more recent v3.8/v4.0 output.

(c) - Like the netpbm tooling there are probably ways to explicitly 
control gamma handling. Output and compare jpgs. Write linear output 
files and compare via linear gamma options...

(d) - Yes. Even if the sRGB block being there and not the 'reason' for 
the pink shift - how are we getting the grey-ish general background 
where both input images are completely black? My first guess is the 
actual pink/grey shift is a bug in 'compare' but, who knows. It might be 
some other reason like compare showing differences in value results as 
offsets from 50% grey or something, though the red differences within 
the sphere sweep I think right! (alpha channel assumptions in compare 
or?) No clue.

Post a reply to this message

On 3/5/22 05:45, William F Pokorny wrote:
> (d) - Yes. Even if the sRGB block being there and not the 'reason' for 
> the pink shift - how are we getting the grey-ish general background 
> where both input images are completely black? My first guess is the 
> actual pink/grey shift is a bug in 'compare' but, who knows. It might be 
> some other reason like compare showing differences in value results as 
> offsets from 50% grey or something, though the red differences within 
> the sphere sweep I think right! (alpha channel assumptions in compare 
> or?) No clue.

OK. Still busy with other things, but had the bright idea to compare an 
image with itself and see what 'compare' does when images match. It's 
not doing a strict difference compare on the channel values (as you 
probably know). In other words, it always creates a faded image where 
things compare.

Further figured out it's using a high light color where things don't 
compare (to some threshold maybe? Per channel? Grey Composite?) and this 
happens to default to red. This why the missing red differences are 
showing up as I happen to expect the channel differences to show up on a 
channel difference compare.

So, compare fine for what it does. In fact, it looks to have some cool 
options and relatively easy defaults. It just isn't doing what I 
expected for image comparisons.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/5/22 05:45, William F Pokorny wrote:
> > (d) - Yes. Even if the sRGB block being there and not the 'reason'

if you ever recall why, please tell.  now that I know about it, I do wonder what
the rationale/potential benefit is when the chunk is not written.


> > ...
> So, compare fine for what it does. In fact, it looks to have some cool
> options and relatively easy defaults. It just isn't doing what I
> expected for image comparisons.

(sorry about red-red colour clash)  the excellent documentation
('/usr/doc/ImageMagick-6/www/' on my box) uses '#rrggbb' notation but you can
use the the X ('rgb.txt') colour names (like!).  re the faded image, if you
supply both lowlight and highlight colour options only the actual differences
show.

somewhat unrelated, I cannot understand why the difference in Y-axis translates,
is this a POV-Ray bug?  (attached, thank you)


regards, jr.

Post a reply to this message

Attachments:
Download 'xlates.tar.xz.dat' (2 KB)

"jr" <cre### [at] gmailcom> wrote:
> ...
> somewhat unrelated, I cannot understand why the difference in Y-axis translates,
> is this a POV-Ray bug?  (attached, thank you)

uh, oh, belay that..  found my thinking cap.  :-)


regards, jr.

Post a reply to this message

On 3/6/22 08:35, jr wrote:
> if you ever recall why, please tell.  now that I know about it, I do wonder what
> the rationale/potential benefit is when the chunk is not written.

OK. Found my related debugging newsgroup post.

Much more detail on the v3.8/v4.0 sRGB block issue being written nor not 
can be found in a Mar 20, 2021 post to povray.beta-test. "File png/ppm 
gamma issues. v3.8." See:

http://news.povray.org/povray.beta-test/thread/%3C60563168%241%40news.povray.org%3E/

or

Message: <60563168$1@news.povray.org>

The thread started after questions from ingo related to pgm/ppm file 
output. I stumbled across the png sRGB block issue during that work. 
Look for "---  png gamma issue.' for the parts of the original post 
related to png input/output.

The bug amounts to POV-Ray not being consistent with 'itself' with 
respect to png sRGB handling.

Looks like in reading the thread again I never did the detailed work to 
figure out what all changed to cause the sRGB block to be dropped on 
default writes though 'srgb' is our default output/input png gamma 
handling profile.

I'd spent many days understanding what was happening in detail and 
didn't want to spend hours to days more trying to run down the commit(s) 
where the post v3.7 srgb png output file gamma handling broke. I just 
fixed the issue - which explains why I couldn't remember the detailed 
causes(a)!

(a) This time it might not have been my old failing brain! Of course, I 
didn't remember I hadn't dug enough to determine exactly how we got into 
the buggy state... ;-)

I know clipka worked quite a bit on the image file handling code over 
years while working toward v3.71/v3.8(v4.0). It was likely broken 
sometime during those changes.

Aside: Because broken v3.71/v3.8/v4.0 versions do still write a gamma 
2.2 chunk, the differences visually be will be small / hard to 'see' - 
they're at the dark end. This probably why the bug went years not 
getting picked up. It's one of those insidious subtle bugs that cause 
confusing / hair pulling results when they do bite.

Bill P.

Post a reply to this message

William F Pokorny <ano### [at] anonymousorg> wrote:

> Aside: Because broken v3.71/v3.8/v4.0 versions do still write a gamma
> 2.2 chunk, the differences visually be will be small / hard to 'see' -
> they're at the dark end. This probably why the bug went years not
> getting picked up. It's one of those insidious subtle bugs that cause
> confusing / hair pulling results when they do bite.

Interesting.

Do you think there's a shell command that will output some unique result to
distinguish between the two types of gamma encoded png files?

If there is, maybe someone can demonstrate how to run a pre-parse/render command
to write that to a file, and then have a macro parse that and set the png read
parameters to properly/consistently adapt to however any given file is written?
(perhaps sending a message to the debug stream to ID which gamma type the file
is...)

I see that as being the best workaround for the time being.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/6/22 08:35, jr wrote:
> > if you ever recall why, ...
>
> OK. Found my related debugging newsgroup post.
>
> Much more detail on the v3.8/v4.0 sRGB block issue being written nor not
> can be found in a Mar 20, 2021 post to povray.beta-test. "File png/ppm
> gamma issues. v3.8." See:
> [snip]

thanks (very much).  so recent, yet, I've no memory of reading the thread.  :-(

> ...
> Aside: Because broken v3.71/v3.8/v4.0 versions do still write a gamma
> 2.2 chunk, the differences visually be will be small / hard to 'see' -
> they're at the dark end. This probably why the bug went years not
> getting picked up. It's one of those insidious subtle bugs that cause
> confusing / hair pulling results when they do bite.

guessing that you "look closer" than I do.  also, I "misuse" gamma perhaps,
mixing sRGB light_source and rgb[ft] colours routinely, bad either the result is
ok-ish or my eyes are not up to it (anymore) :-).


regards, jr.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> ... It's one of those insidious subtle bugs that cause
> confusing / hair pulling results when they do bite.

may have run into one those with 'povr' yesterday.  been using it while playing
with the USaF (see p.b.a) code since it's quicker to render.  all ok for the 500
frame tests, but when I went to do a "final" render, it crashed
("uncategorized", from memory) at the 3048th frame.  (sorry, bearer of bad news
and all that)


regards, jr.

Post a reply to this message

On 3/13/22 04:34, jr wrote:
> hi,
> 
> William F Pokorny <ano### [at] anonymousorg> wrote:
>> ... It's one of those insidious subtle bugs that cause
>> confusing / hair pulling results when they do bite.
> 
> may have run into one those with 'povr' yesterday.  been using it while playing
> with the USaF (see p.b.a) code since it's quicker to render.  all ok for the 500
> frame tests, but when I went to do a "final" render, it crashed
> ("uncategorized", from memory) at the 3048th frame.  (sorry, bearer of bad news
> and all that)
> 
> 
> regards, jr.
> 

This with the Real_Time_Capture=true|false (+-rtc) capability for the 
real time rendering mode (.pfm output) or with traditional animation?

Is the fail easily repeatable?

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> This with the Real_Time_Capture=true|false (+-rtc) capability for the
> real time rendering mode (.pfm output) or with traditional animation?

regular animation, running from an ini.


> Is the fail easily repeatable?

not sure, happened late in the animation[*], had no stomach to try + repeat,
sorry.  the code in question is the 'usaf' animation posted same time (March
13th).

[*] as I wrote, I was v surprised because 'povr' run the (shorter) test segments
without hassle, and fast.


regards, jr.

Post a reply to this message

On 3/17/22 02:04, jr wrote:
>> Is the fail easily repeatable?
> not sure, happened late in the animation[*], had no stomach to try + repeat,
> sorry.  the code in question is the 'usaf' animation posted same time (March
> 13th).

No problem. The flaky issues / bugs are always the hard ones to run down.

Did you continue the animation successfully starting from frame 3048?
Or, is it an animation where each frame depends on information from 
previous frames(s) such that starting at frame 3048 not possible?

I might try running valgrind looking for memory leaks with your 
particular animation as a shot in the dark. But, that's not something 
I'll get to for a while.

Wondering too if the regular animation mechanisms can develop the memory 
bubbles seen with RTR. My guess is the frame rates with regular 
animation always too slow for it to happen - and as a light user of 
regular animation I've never noticed a (povms) memory bubble effect but, 
never gone looking for it, so who knows.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/17/22 02:04, jr wrote:
> >> Is the fail easily repeatable?
> > not sure, ...
> No problem. The flaky issues / bugs are always the hard ones to run down.

took copy of code posted + tried again, several times, every time the run (now)
crashes at frame 2800; have tried to remember which if any changes I made, but
can only think of quality + dimensions in ini file.  attached excerpt from
'povr' build-log, libraries in use, and the animation state at the time.



> Did you continue the animation successfully starting from frame 3048?
> Or, is it an animation where each frame depends on information from
> previous frames(s) such that starting at frame 3048 not possible?

no, did not.  simply switched to POV-Ray proper, wanted "it" (the anim) out of
my hair :-).


> I might try running valgrind looking for memory leaks with your
> particular animation as a shot in the dark. But, that's not something
> I'll get to for a while.

running with 480x360 and aa threshold .3 should do.


> Wondering too if the regular animation mechanisms can develop the memory
> bubbles seen with RTR. My guess is the frame rates with regular
> animation always too slow for it to happen - and as a light user of
> regular animation I've never noticed a (povms) memory bubble effect but,
> never gone looking for it, so who knows.

sorry, no ideas at all.  (it's as if the 'tmp_' dictionary is not updated in
time for next access)


regards, jr.

Post a reply to this message

Attachments:
Download 'frame_2800.txt' (4 KB)

On 3/17/22 11:19, jr wrote:
> Degenerate cylinder, base point = apex point.
> Fatal error in parser: Uncategorized error.
> Render failed

The three lines above come from a test during parsing while additional 
internal data for the cone/cylinder is being done using a function 
within cone.cpp. The tests is finding that the base and apex are <1e-10 
(a) apart - and throwing an error rather than issuing a parse error via 
message(b). The same code exists in POV-Ray proper v3.7 onward and povr 
(c).

(a) - The cone code hasn't been reworked as yet in povr to remove 
'EPSILON' use. The test value there should be ~4.4e-8 or povr's 
gkMinIsectDepthReturned. FWIW.

(b) - The particular test could be done completely inside the parser I 
think. Not sure why it's done in the cone/cylinder shape code.

(c) - There has long been a TODO note in the code suggesting that 
particular throw should be a 'possible error' instead - with execution 
continuing. I don't agree with the TODO note. Where base and apex are 
too close, it's an error in cone/cylinder specification.

---
The puzzle for me at the moment is I have no idea why those parsed base 
and apex vectors would be different between any versions of POV-Ray.

Best guess. It's due some indirect difference where I've moved to 
'double' from 'float' in povr and the previous 'snap' of the vector 
components to floats rounded length 'up' in a way not tripping the 
throw. In other words, values previously rounded such that the base to 
apex length is 'long enough'.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 3/17/22 11:19, jr wrote:
> > Degenerate cylinder, base point = apex point.
> > Fatal error in parser: Uncategorized error.
> > Render failed
>
> The three lines above come from a test during parsing while additional
> internal data for the cone/cylinder is being done using a function
> within cone.cpp. The tests is finding that the base and apex are <1e-10
> (a) apart - and throwing an error rather than issuing a parse error via
> message(b). The same code exists in POV-Ray proper v3.7 onward and povr
> (c).
>
> (a) - The cone code hasn't been reworked as yet in povr to remove
> 'EPSILON' use. The test value there should be ~4.4e-8 or povr's
> gkMinIsectDepthReturned. FWIW.
>
> (b) - The particular test could be done completely inside the parser I
> think. Not sure why it's done in the cone/cylinder shape code.
> ...
> The puzzle for me at the moment is I have no idea why those parsed base
> and apex vectors would be different between any versions of POV-Ray.

thank you v much.  the thing I don't understand is why, when every increment is
a whole unit, we'd get into a situation where (the value of) Epsilon would
matter.  and while the above reasoning does make sense, I can't shake the
feeling that, underneath, it's a dictionary "thing"/problem.


regards, jr.

Post a reply to this message

(correction)

"jr" <cre### [at] gmailcom> wrote:
> ...
> and while the above reasoning does make sense, I can't shake the
> feeling that, underneath, it's a dictionary "thing"/problem.

sorry, meant to write that the actual error is, I feel, misleading, and perhaps
is a dictionary/macro "thing".

<https://en.wikipedia.org/wiki/Allegory_of_the_cave>


regards, jr.

Post a reply to this message

On 3/18/22 03:32, jr wrote:
> (correction)
> 
> "jr" <cre### [at] gmailcom> wrote:
>> ...
>> and while the above reasoning does make sense, I can't shake the
>> feeling that, underneath, it's a dictionary "thing"/problem.
> 
> sorry, meant to write that the actual error is, I feel, misleading, and perhaps
> is a dictionary/macro "thing".
> 
> <https://en.wikipedia.org/wiki/Allegory_of_the_cave>
> 

Played a little with your animation this morning and see similar parse 
time fails. Continuing from the failing frame (ignoring whether this 
correct for the scene for now) always seems to work. So, it's not the 
particular frame calculations as you suggested.

The fails after re-starts from previous failing frames come faster and 
faster the larger the size of the usaf_items.csv file. This hints at a 
race condition where we are, perhaps, picking up base or apex twice as 
base and apex.

Debugging will take time I don't have at the moment. I'll capture the 
current work in a debug directory with the intent to come back to it later.

The good news is the fails are somewhat reproducible - though it's a bit 
of a pain to reach the 'fail more frequently' state.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> ...
> Played a little with your animation this morning and see similar parse
> time fails.

glad it reproduces, and not result of compile-time choices/optimisation.

sorry for late-ish reply, re-ran the anim a number of times last night + this
morning.


> Continuing from the failing frame (ignoring whether this
> correct for the scene for now) always seems to work. So, it's not the
> particular frame calculations as you suggested.
>
> The fails after re-starts from previous failing frames come faster and
> faster the larger the size of the usaf_items.csv file. This hints at a
> race condition where we are, perhaps, picking up base or apex twice as
> base and apex.
> Debugging will take time I don't have at the moment. I'll capture the
> current work in a debug directory with the intent to come back to it later.

thanks.  adding some "data points" which, hopefully, will be of use then.

started with clean copy as posted, and in ini dialled down AA threshold (.3) and
commented out the 'all_file' line.  rendering then fails, with error, after
image 2799, repeatable[*] (take "similar" to mean you see different frame#?
anyway, expect it to stay the same).  I then switched verbose + logging off in
the scene, same error but after image 2876, also repeatable.  next tried with
verbose true but no logging, same error, this time after frame 2895.  then it
got .. tasty :-).  ran with verbose false and logging enabled, and got (out of
five runs) two successes and three segfaults (all past frame 3500, but different
#s).
  povray2[664477]: segfault at 25 ip 00000000004c733

[*] each test done twice or more.

I'd be interested to know whether your's too crashes.


> The good news is the fails are somewhat reproducible - though it's a bit
> of a pain to reach the 'fail more frequently' state.

not done it, but if state{curr} and items{point} agree, you should be able to
restart with '+kfi' etc.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:
> ...
> thanks.  adding some "data points" which, hopefully, will be of use then.

another of-interest-perhaps point.  added a '#debug concat()' as first line of
the '__mkItems()' macro, outputting the macro arguments (of which index is not
o/wise used in the code).  now get a different error, at frame 2735: "Expected
'numeric expression', 'string identifier' found".  hypothesis - macro gets
wrong/corrupted set of args, the first of which (i_) happens to look like a
string?  anyway, intriguing (and annoying :-)).


regards, jr.

Post a reply to this message

On 3/19/22 09:41, jr wrote:
> then it got .. tasty 😄.  ran with verbose false and logging enabled, and got (out
of
> five runs) two successes and three segfaults (all past frame 3500, but different
> #s).
>    povray2[664477]: segfault at 25 ip 00000000004c733
> 
> [*] each test done twice or more.
> 
> I'd be interested to know whether your's too crashes.

Looking at this issue again after a year! (Sorry)

I can now see the exact re-producible fails you'd mentioned. I changed 
the cylinder source code so it simply writes the duplicate apex base 
vectors rather than issue a throw(a). Starting always from a saved case 
which reached frame 3000. I run repeatedly and it is always the same 
three fails!

I, like you, have had other fail signatures, but it looks like if we 
start with certain fixed conditions the fail points are indeed the same.
This perhaps enough information for me to insert a break point.

I've tried a few other things shooting in the dark and like you get as 
often as not 'other' odd results.

---

At the moment I am trying to get your seg fault signature above using a 
debug compile having symbols - where the core file generated should be 
useful.

You didn't get this fail every time, and so far I've not gotten a 
segfault. I'm just running it and running it again for a while with 
verbose false and logging enabled. We'll see.

Bill P.

Post a reply to this message

On 3/24/23 11:39, William F Pokorny wrote:
> At the moment I am trying to get your seg fault signature above using a 
> debug compile having symbols - where the core file generated should be 
> useful.
> 
> You didn't get this fail every time, and so far I've not gotten a 
> segfault. I'm just running it and running it again for a while with 
> verbose false and logging enabled. We'll see.

No segfaults of the differing type you saw.

I did get two different Parse Error signatures early. One for an array 
subscript out of range and another via the include of parse_fore.tmp 
attempting to access an uninitialized array element. Neither made sense 
to me.

Then the blasted povr code proceeded to run cleanly both partial runs 
and full ones for 6 passes.

I went back to the original failing set up starting at frame 3000 with 
both verbose and log frames on - which had been failing at the same 
'spots' for me - and at least the most recent pass decided other spots / 
cylinders should now fail.

Only thing that's clear is what was clear before - we have no problems 
until the frame counts are 2500, 3000 or more.

Anyhow. Answering the question you asked a year ago. I don't see the 
segfaults you saw with verbose off and logging on. Mostly things worked 
in this configuration...

Future posts on this povr only problem, I'll put in a new thread 
somewhere. Probably user.patches. This thread got really long and became 
'twelve ways' unrelated to the original topic!

Bill P.

Post a reply to this message