POV-Ray : Newsgroups : povray.binaries.images : POVEarth still far away... Server Time: 24 Sep 2020 18:47:48 GMT
  POVEarth still far away... (Message 11 to 18 of 18)  
<<< Previous 10 Messages Goto Initial 10 Messages
From: Jörg "Yadgar" Bleimann
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 16:37:56
Message: <5f26ebe4$1@news.povray.org>
On 02.08.20 16:23, jr wrote:
> hi,

> there's an option for that.
> "All_File=true
> Echo all debug, fatal, render, statistic, and warning text to ALLTEXT.OUT"
>
<http://wiki.povray.org/content/Reference:Text_Output_Options#Directing_Text_Streams_to_Files>
> 
> alternatively, on Linux, you can do something like:
> 
>    $ povray myini 2> mylog
> 
> to capture all povray's text output.
> 

I did the latter... but nothing really interesting had been captured, 
just some one million lines of vertex generation messages from the 
#warning stream - nothing else, no fatal, render, debug or statistic 
streams! Strange?

See you in Khyberspace!

Yadgar


Post a reply to this message

From: jr
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 18:25:00
Message: <web.5f27040d93bb714f4d00143e0@news.povray.org>
hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> On 02.08.20 16:23, jr wrote:
> > there's an option for that.
> > "All_File=true
> > Echo all debug, fatal, render, statistic, and warning text to ALLTEXT.OUT"
> >
<http://wiki.povray.org/content/Reference:Text_Output_Options#Directing_Text_Streams_to_Files>
> >
> > alternatively, on Linux, you can do something like:
> >
> >    $ povray myini 2> mylog
> >
> > to capture all povray's text output.
> >
>
> I did the latter... but nothing really interesting had been captured,
> just some one million lines of vertex generation messages from the
> #warning stream - nothing else, no fatal, render, debug or statistic
> streams! Strange?

I assume you're not using any of the other 'text output options' to suppress
output on some streams?  the 'verbose' option too is set 'on', I assume?
thereafter (afaik), there's only what Bald Eagle wrote, ie adding '#debug
concat()'s to the code.

also, if you want to use shell redirection and see the output as it's written,
you could use the 'tee' utility, eg:

  $ povray myini 2>&1 | tee mylog


regards, jr.


Post a reply to this message

From: Jörg "Yadgar" Bleimann
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 18:52:37
Message: <5f270b75$1@news.povray.org>
Hi(gh)!

On 02.08.20 16:18, Bald Eagle wrote:

> Hmm.   That's pretty weird.
> What's your OS & povray version?

OS: Debian Linux 8.11
POV-Ray: 3.7.0.unofficial

> Have you tried running a different version or changing the #version statement at
> the top of the file?

I tried adding #version 3.6; - same as always before, memory access 
error somewhere between a=18 and a=22.

> What about using #for instead of #while ?

I tried this also - memory access error at a=88.

> Stepping through the values by 2's ?

...is effectively by 3's (why?) - memory access error at a=171.

> If you can post an easy-to-follow version (indents, some comments) of your full
> code in the text file section, maybe I can spot something.

I did so, look at p.t.s-f

> Then everything that goes to the debug stream will go to the file as well.
> http://www.povray.org/documentation/view/3.6.1/222/

This produced just a seemingly endless rush of vertex numbers (from 0 to 
999999), without any hint at the reason for the memory access error.

See you in Khyberspace!

Yadgar


Post a reply to this message

From: Bald Eagle
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 19:55:01
Message: <web.5f2719f193bb714f1f9dae300@news.povray.org>
=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> Hi(gh)!
>
> On 02.08.20 16:18, Bald Eagle wrote:

> > Stepping through the values by 2's ?
>
> ...is effectively by 3's (why?) - memory access error at a=171.

To see if processing less data got you through.


> > Then everything that goes to the debug stream will go to the file as well.
> > http://www.povray.org/documentation/view/3.6.1/222/
>
> This produced just a seemingly endless rush of vertex numbers (from 0 to
> 999999), without any hint at the reason for the memory access error.

Yeah, well - whatever you send to the debug stream gets written there, and
whatever you don't - doesn't.

I edited your file for (my) clarity, added some time stamps, and ran it.
Near the end of your file you have a missing closing parentheses:

#write (ES  "  finish{ F_Earthslice }\n")
#write (ES, "    }\n")   // <--------- missing closing )

Once I fixed that, it ran just "fine".
It's slow as hell and uses a LOT of memory, which is likely your problem.  You
probably run OUT of memory somewhere.

-------------------- STARTING NESTED LOOP 1 --------------------
NESTED LOOP 1 Time elapsed = 62.536 sec.
-------------------- STARTING NESTED LOOP 2 --------------------
NESTED LOOP 2 Time elapsed = 19.933 sec.
-------------------- STARTING NESTED LOOP 3 --------------------
NESTED LOOP 3 Time elapsed = 118.422 sec.
-------------------- STARTING LOOP 1 --------------------
LOOP 1 Time elapsed = 20.327 sec.
-------------------- STARTING LOOP 2 --------------------
LOOP 2 Time elapsed = 18.262 sec.
-------------------- STARTING NESTED LOOP 4 --------------------
NESTED LOOP 4 Time elapsed = 33.183 sec.
-------------------- STARTING LOOP 3 --------------------
LOOP 3 Time elapsed = 47.177 sec.
Total Time elapsed = 319.840 sec.

It takes 5 minutes to process your one (not very large) image file.
Parse Time:       0 hours  5 minutes 20 seconds (320.167 seconds)

So, I would suggest that you take a hard look at your code, see what is NEEDED
and what is not, trim the fat, and then optimize your code to use less than 7+
loops to construct a mesh from the pixel data.

You should be able to skip over performing certain operations using #if (value =
0) or #if (value = finalvalue-1) or #if (value = finalvalue)

That 3rd nested loop is an absolute killer.
You do a lot of the same calculations many MANY times.
Pre-calculate equation terms before all of your conditional statements, and then
you not only save time but also make the expressions much shorter and easily
read.

Don't tackle such large data sets until you have the basic code at least
somewhat optimized.   Crop out a small portion of the original image as a test
area, or use a larger step size to skip over some percentage of the data.

You're processing over a million tokens in theis scene with that image, so you
likely just don't have the system resources to handle the array data that you're
piling up.   If you can just use temporary variable that can get redeclared in
the next step of the loop, you might be just fine.


Post a reply to this message

From: Bald Eagle
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 20:05:00
Message: <web.5f271b8593bb714f1f9dae300@news.povray.org>
Also, for nested loop 3, you're using the eval_pigment MACRO from an include
file "functions.inc"

All that does is:
#macro eval_pigment(pigm, vec)
    #local fn = function { pigment { pigm } }
    #local result = (fn(vec.x, vec.y, vec.z));
    result
#end

Somehow this magically returns a vector quantity, rather than a scalar.  :O

But the point is, that maybe declaring the function at the beginning of the
scene and just evaluating it when you need to without a lot of intermediate
steps and storage variables might cut down on memory usage and speed things up a
bit.


Post a reply to this message

From: Jörg "Yadgar" Bleimann
Subject: Re: POVEarth still far away...
Date: 2 Aug 2020 21:53:31
Message: <5f2735db$1@news.povray.org>
Hi(gh)!

On 02.08.20 21:54, Bald Eagle wrote:

> I edited your file for (my) clarity, added some time stamps, and ran it.
> Near the end of your file you have a missing closing parentheses:
> 
> #write (ES  "  finish{ F_Earthslice }\n")
> #write (ES, "    }\n")   // <--------- missing closing )

Okay, that's the ticket! Thank you VERY much... that's a whole asteroid 
off my heart! What a stupid little mistake...

Probably the seemingly endless #write sequence led to memory overload...

> 
> Once I fixed that, it ran just "fine".
> It's slow as hell and uses a LOT of memory, which is likely your problem.  You
> probably run OUT of memory somewhere.

The original version by Melody was even much, much slower - generating a 
3601 by 3601 mesh took some cool 200 days! My version does it in two and 
a half hours...

> That 3rd nested loop is an absolute killer.
> You do a lot of the same calculations many MANY times.
> Pre-calculate equation terms before all of your conditional statements, and then
> you not only save time but also make the expressions much shorter and easily
> read.

I don't understand what you mean - the very formula by which the sums of 
the vector cross products are calculation depends on the position of the 
respective vertex, which are divided into nine different cases: lower 
left, lower right, upper left and upper right corners, lower, left, 
right and upper side, center. I cannot pre-calculate anything! At least 
not without the risk of messing up the code completely... never change a 
running system!

> 
> Don't tackle such large data sets until you have the basic code at least
> somewhat optimized.   Crop out a small portion of the original image as a test
> area, or use a larger step size to skip over some percentage of the data.
> 
> You're processing over a million tokens in theis scene with that image, so you
> likely just don't have the system resources to handle the array data that you're
> piling up.   If you can just use temporary variable that can get redeclared in
> the next step of the loop, you might be just fine.
> 

Before that parenthesis went lost, it worked fine... probably slower 
than necessary, but it worked!

See you in Khyberspace!

Yadgar


Post a reply to this message

From: Bald Eagle
Subject: Re: POVEarth still far away...
Date: 3 Aug 2020 00:15:00
Message: <web.5f2756a293bb714f1f9dae300@news.povray.org>
> Okay, that's the ticket! Thank you VERY much... that's a whole asteroid
> off my heart! What a stupid little mistake...
>
> Probably the seemingly endless #write sequence led to memory overload...

Well - it's those damned pesky little parentheses again.


> The original version by Melody was even much, much slower - generating a
> 3601 by 3601 mesh took some cool 200 days! My version does it in two and
> a half hours...

OMG - that's ridiculous.
Use what you have, but start experimenting with making a version that's MUCH
faster.


> I don't understand what you mean

Just casually look at that block of code.
How many individual times do you calculate "xdim+(b-1)"?
Using that as a starting point, how many arithmetic operations can you "roll
into one"?

I cannot pre-calculate anything!
Yes you can.  Look, think, TRY.
Sometimes small things, sometime larger patterns reveal themselves.


> Before that parenthesis went lost, it worked fine... probably slower
> than necessary, but it worked!

Dude - if you know you can use a table saw, why rip an 8ft board by hand?
I'm using a 21600 x 10800 pixel heightfield for an _isosurface_, and I _render_
that in minutes, not hours....
How are you going to process _multiple_ tiles?   :O


Post a reply to this message

From: Alain Martel
Subject: Re: POVEarth still far away...
Date: 3 Aug 2020 14:36:25
Message: <5f2820e9$1@news.povray.org>
Le 2020-08-02 à 06:54, Jörg "Yadgar" Bleimann a écrit :
> Hi(gh)!
> 
> I now have run mesh2writer.pov in various RAM configurations including 
> only one 4 GB strip or one 8 GB strip in slot 1 - but always I got the 
> memory access error in the 22nd outer iteration (at 1000 by 1000 data 
> points; the number of the inner iteration varies wildly!) of the last 
> nested loop:
> 
> #if (!(tilename="n80e014" | tilename="n80e016" | tilename="n80e017"))
>    #declare i=0;
>    #write (ES, "  texture_list")
>    #write (ES, "  {\n")
>    #write (ES, concat("    ",str(NumTextures, 1, 0),"\n"))
>    #declare a=0;
>    // #declare c=0;
>    #while (a<ydim)                    // outer loop
>      #declare b=0;
>      #while (b<xdim)                    // inner loop
>        #declare C_Texture = eval_pigment(P_Texture, 
> <(0.5+b)*(1/(xdim-1)), (0.5+a)*(1/(ydim-1)), 1>);
>        #warning concat("Iteration ", str(a, 1, 0), "-", str(b, 1, 0))
>        #write (ES, "    texture\n")
>        #write (ES, "    {\n")
>        #write (ES, "      pigment\n")
>        #write (ES, "      {\n")
>        #write (ES, concat("       color rgb <",vstr(3, C_Texture, ",", 
> 1, 7),">\n"))
>        #write (ES, "      }\n")
>        #write (ES  "     finish{ F_Earthslice }\n")
>        #write (ES, "    }\n"
>        // #declare c=c+1;
>        #declare b=b+1;
>      #end
>      #declare a=a+1;
>    #end
>    #write (ES, "  }\n")
> #end
> 
> // end of code
> 
> I just can't figure out what should be wrong with this code - or is it 
> just a bug in POV-Ray 3.7?
> 
> See you in Khyberspace!
> 
> Yadgar

Given that the error happens during a write operation to the disk, I 
suspect that the actual error may be related to disk access issue rather 
than a RAM issue.
What is the size of the file when the error happens and how much disk 
space is available?
How the current file size relate to your cluster size ?

Try creating a dummy file before the run. For example, save a small 
image of about 8 to 10K. Does that cause the timing of the error to change ?


Alain


Post a reply to this message

<<< Previous 10 Messages Goto Initial 10 Messages

Copyright 2003-2008 Persistence of Vision Raytracer Pty. Ltd.