Hi(gh)!

I looked up texture_list and texture indices... and finally came up with 
a solution, assigning each two adjacent triangles one texture. But... 
long years in a crowded, badly ventilated case took their toll on system 
health. Both under Linux and Windows POV-Ray 3.7 crashed with a memory 
access error when trying to generate a full-resolution mesh2 of the 
Tbilisi square-degree quadrangle, most likely due to chronically 
overheated and thus faulty RAM. I might thoroughly clean my case and its 
innards from dust and then run memtest, but whatever the result, it 
might cost me at least 100 euros... if not much more, depending how much 
of my 24 GiB RAM is damaged. The worst case would be having to buy a 
complete new system... this probably would take me several years of saving!

Reality hurts! If I only could escape from it... building Khyberspace 
entirely in my brain?

See you there,

Yadgar

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> Hi(gh)!
>
> ... The worst case would be having to buy a
> complete new system... this probably would take me several years of saving!

surprised by this.  is there no "flourishing" second-hand market for computers
and or components in Germany?  there must be companies replacing equipment "on
schedule", just like with their (fleet) cars.  you just need to find out who are
the re-sellers.


regards, jr.

Post a reply to this message

Hi(gh)!

On 15.07.20 16:09, jr wrote:
> hi,
> 
> =?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
>> Hi(gh)!
>>
>> ... The worst case would be having to buy a
>> complete new system... this probably would take me several years of saving!
> 
> surprised by this.  is there no "flourishing" second-hand market for computers
> and or components in Germ  there must be companies replacing equipment "on
> schedule", just like with their (fleet) cars.  you just need to find out who are
> the re-sellers.
> 
> 
> regards, jr.
> 
any?

Yes, it is... but for doing POVEarth (or even just Khyberspace!) with 
reasonable resources, a system needs to be state-of-the-art, something 
like an AMD Ryzen Threadripper with 256 GiB worth of RAM and harddisk 
storage in the two-digit terabyte range - far beyond my reach, unless I 
break a lotto jackpot!

It's possibly more likely that one day, I find some ready-made pre-war 
Khyberspace (i. e. a virtual Afghanistan) somewhere on the Internet and 
just join it as an ordinary "tourist"...

See you in Khyberspace!

Yadgar

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> Yes, it is... but for doing POVEarth (or even just Khyberspace!) with
> reasonable resources, a system needs to be state-of-the-art, something
> like an AMD Ryzen Threadripper with 256 GiB worth of RAM and harddisk
> storage in the two-digit terabyte range - far beyond my reach, unless I
> break a lotto jackpot!

I think he meant just to buy some more RAM, second-hand.

I used to work for an electronics surplus place, and we could have probably
filled a pallet with memory sticks of every description.
We could have probably filled several pallets with the hard disks.

You can buy new 1TB HDD for ~35 USD (FRN) or 8 TB for $150
Why you'd need that much, I have no idea.


You can check whatever the equivalent of Ebay and Craigslist for computer
listings, look for used or damaged systems at yard sales, flea markets, moving
sales, estate sales - you really just never know what someone is willing to just
give away for a few dollars.

Post a reply to this message

Hi(gh)

(two weeks later...)

Am 15.07.2020 um 19:19 schrieb Bald Eagle:

> I think he meant just to buy some more RAM, second-hand.
>
> I used to work for an electronics surplus place, and we could have probably
> filled a pallet with memory sticks of every description.
> We could have probably filled several pallets with the hard disks.
>
> You can buy new 1TB HDD for ~35 USD (FRN) or 8 TB for $150
> Why you'd need that much, I have no idea.
>
>
> You can check whatever the equivalent of Ebay and Craigslist for computer
> listings, look for used or damaged systems at yard sales, flea markets, moving
> sales, estate sales - you really just never know what someone is willing to just
> give away for a few dollars.
>

Meanwhile, I installed Memtest86+ and run it now since almost nine 
hours... but still no ram error has been found! And I worked out more 
precisely where that memory access error with mesh2writer.pov occurs: 
always in the 22th iteration of the outer part of the last nested loop 
(currently, I don't have the code at hand, as I plan to run Memtest86+ 
until tomorrow morning - I write this message from my old laptop).

If even tomorrow no RAM error would have been found, there would be no 
way but running my test version of mesh2writer.pov (reduced resolution, 
1000 by 1000 rather than the original 3601 by 3601 data points) several 
times with another out of four RAM strips left out each time...

Until then, I'll have some fun with my Commodore 64 recently 
reactivated... time to build a directory for my 150+ diskettes!

See you in Khyberspace!

Yadgar

Post a reply to this message

Hi(gh)!

I now have run mesh2writer.pov in various RAM configurations including 
only one 4 GB strip or one 8 GB strip in slot 1 - but always I got the 
memory access error in the 22nd outer iteration (at 1000 by 1000 data 
points; the number of the inner iteration varies wildly!) of the last 
nested loop:

#if (!(tilename="n80e014" | tilename="n80e016" | tilename="n80e017"))
   #declare i=0;
   #write (ES, "  texture_list")
   #write (ES, "  {\n")
   #write (ES, concat("    ",str(NumTextures, 1, 0),"\n"))
   #declare a=0;						
   // #declare c=0;
   #while (a<ydim)					// outer loop
     #declare b=0;
     #while (b<xdim)					// inner loop	
       #declare C_Texture = eval_pigment(P_Texture, 
<(0.5+b)*(1/(xdim-1)), (0.5+a)*(1/(ydim-1)), 1>);
       #warning concat("Iteration ", str(a, 1, 0), "-", str(b, 1, 0))
       #write (ES, "    texture\n")
       #write (ES, "    {\n")
       #write (ES, "      pigment\n")
       #write (ES, "      {\n")
       #write (ES, concat("	   color rgb <",vstr(3, C_Texture, ",", 1, 
7),">\n"))
       #write (ES, "      }\n")
       #write (ES  "	 finish{ F_Earthslice }\n")
       #write (ES, "    }\n"
       // #declare c=c+1;
       #declare b=b+1;
     #end
     #declare a=a+1;
   #end
   #write (ES, "  }\n")
#end

// end of code

I just can't figure out what should be wrong with this code - or is it 
just a bug in POV-Ray 3.7?

See you in Khyberspace!

Yadgar

Post a reply to this message

Yadgar Bleimann wrote:

> always I got the
> memory access error in the 22nd outer iteration (at 1000 by 1000 data
> points; the number of the inner iteration varies wildly!) of the last
> nested loop:

The fact that you're getting a very reproducible error in a highly specific
point in the code, tells me that the problem isn't with your RAM, it's with
either the input to your code, or a calculation in your code, which _might_ (or
might not) be triggering a bug of some sort.

by "the 22nd outer iteration ... of the last nested loop", I prsume that you
mean when a = 21?

>    #while (a<ydim)     // outer loop

Here it looks to me that you have a calculation that then gets passed to a
function that operates on image data.  I just have the feeling that this is the
line that triggers the error.   Have you tried commenting that out and running
the code to see if it completes?
If it does, then I would look at the values very carefully, and maybe play some
games like "#(if a = 21)..."
But first I'd also send your values to the #debug stream and perhaps a log file.
Look at the image in P_Texture.  What size is it?  Have you spreadsheeted out
your x, y values to see if they make sense?   What happens at a = 21?
Why is z = 1 instead of 0?
Have you tried using a different image file or running it through a different
piece of software to see if it changes anything? (just give the brightness or
contrast a slight tweak to change the image and save a new version).

>        #declare C_Texture = eval_pigment(P_Texture,
> <(0.5+b)*(1/(xdim-1)), (0.5+a)*(1/(ydim-1)), 1>);

I see that you have this finish statement in _every line_.

>        #write (ES  "  finish{ F_Earthslice }\n")

Why not just define that as the default finish and leave it out?

Post a reply to this message

Hi(gh)!

On 02.08.20 14:03, Bald Eagle wrote:

> by "the 22nd outer iteration ... of the last nested loop", I prsume that you
> mean when a = 21?

Technically, it's the 23rd iteration, i. e. when a=22!
I now scaled the original 3601 by 3601 heightfield down to 1000 x 1000 
(instead of just using 1000 x 1000 eval_pigment() tracing points on the 
original heightfield) using GIMP and no interpolation(!) - and got 
exactly the same result: memory access error at a=22!

Then I took a close look at the pixels in that row (as the lines are 
processed in ascending order, this is line 977 (999-22) in the GIMP 
display) - and found nothing suspicious, the first pixel in that line is 
rgb <74, 230, 0>, which should calculate to something like
rgb <0.290196078, 0.901960784, 0> in eval_pigm().

> 
>>     #while (a<ydim)     // outer loop
> 
> Here it looks to me that you have a calculation that then gets passed to a
> function that operates on image data.  I just have the feeling that this is the
> line that triggers the error.   Have you tried commenting that out and running
> the code to see if it completes?

I tried this (and used a uniform color, rgb <0.5, 0.5, 0.5>)... before, 
I also discovered another flaw in the code, res was still set to 3601... 
I corrected this to 1000, which did not change anything.

But now with the uniform color, it's even more absurd: the loop breaks, 
but no longer consistently at a=22, but during the first run at a=19, 
during the second once more at a=22, during the third run at a=20...

> But first I'd also send your values to the #debug stream and perhaps a log file.

How do I do this?

See you in Khyberspace!

Yadgar

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> But now with the uniform color, it's even more absurd: the loop breaks,
> but no longer consistently at a=22, but during the first run at a=19,
> during the second once more at a=22, during the third run at a=20...

Hmm.   That's pretty weird.
What's your OS & povray version?
Have you tried running a different version or changing the #version statement at
the top of the file?

What about using #for instead of #while ?
Stepping through the values by 2's ?
If you can post an easy-to-follow version (indents, some comments) of your full
code in the text file section, maybe I can spot something.


> > But first I'd also send your values to the #debug stream and perhaps a log file.
>
> How do I do this?

You already know how to do:
#write (ES, concat("    color rgb <",vstr(3, C_Texture, ",", 1,
7),">\n"))

So just add a line with
#debug concat("    color rgb <",vstr(3, C_Texture, ",", 1,
7),">\n")


in your command line, you can just add Debug_File=LoopCrash.txt
so that you have something like
+w1024 +h768 +A0.3 Debug_File=LoopCrash.txt

Then everything that goes to the debug stream will go to the file as well.
http://www.povray.org/documentation/view/3.6.1/222/

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> Hi(gh)!
> On 02.08.20 14:03, Bald Eagle wrote:
> ...
> > But first I'd also send your values to the #debug stream and perhaps a log file.
>
> How do I do this?

there's an option for that.
"All_File=true
Echo all debug, fatal, render, statistic, and warning text to ALLTEXT.OUT"
<http://wiki.povray.org/content/Reference:Text_Output_Options#Directing_Text_Streams_to_Files>

alternatively, on Linux, you can do something like:

  $ povray myini 2> mylog

to capture all povray's text output.


regards, jr.

Post a reply to this message

On 02.08.20 16:23, jr wrote:
> hi,

> there's an option for that.
> "All_File=true
> Echo all debug, fatal, render, statistic, and warning text to ALLTEXT.OUT"
>
<http://wiki.povray.org/content/Reference:Text_Output_Options#Directing_Text_Streams_to_Files>
> 
> alternatively, on Linux, you can do something like:
> 
>    $ povray myini 2> mylog
> 
> to capture all povray's text output.
> 

I did the latter... but nothing really interesting had been captured, 
just some one million lines of vertex generation messages from the 
#warning stream - nothing else, no fatal, render, debug or statistic 
streams! Strange?

See you in Khyberspace!

Yadgar

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> On 02.08.20 16:23, jr wrote:
> > there's an option for that.
> > "All_File=true
> > Echo all debug, fatal, render, statistic, and warning text to ALLTEXT.OUT"
> >
<http://wiki.povray.org/content/Reference:Text_Output_Options#Directing_Text_Streams_to_Files>
> >
> > alternatively, on Linux, you can do something like:
> >
> >    $ povray myini 2> mylog
> >
> > to capture all povray's text output.
> >
>
> I did the latter... but nothing really interesting had been captured,
> just some one million lines of vertex generation messages from the
> #warning stream - nothing else, no fatal, render, debug or statistic
> streams! Strange?

I assume you're not using any of the other 'text output options' to suppress
output on some streams?  the 'verbose' option too is set 'on', I assume?
thereafter (afaik), there's only what Bald Eagle wrote, ie adding '#debug
concat()'s to the code.

also, if you want to use shell redirection and see the output as it's written,
you could use the 'tee' utility, eg:

  $ povray myini 2>&1 | tee mylog


regards, jr.

Post a reply to this message

Hi(gh)!

On 02.08.20 16:18, Bald Eagle wrote:

> Hmm.   That's pretty weird.
> What's your OS & povray version?

OS: Debian Linux 8.11
POV-Ray: 3.7.0.unofficial

> Have you tried running a different version or changing the #version statement at
> the top of the file?

I tried adding #version 3.6; - same as always before, memory access 
error somewhere between a=18 and a=22.

> What about using #for instead of #while ?

I tried this also - memory access error at a=88.

> Stepping through the values by 2's ?

...is effectively by 3's (why?) - memory access error at a=171.

> If you can post an easy-to-follow version (indents, some comments) of your full
> code in the text file section, maybe I can spot something.

I did so, look at p.t.s-f

> Then everything that goes to the debug stream will go to the file as well.
> http://www.povray.org/documentation/view/3.6.1/222/

This produced just a seemingly endless rush of vertex numbers (from 0 to 
999999), without any hint at the reason for the memory access error.

See you in Khyberspace!

Yadgar

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> Hi(gh)!
>
> On 02.08.20 16:18, Bald Eagle wrote:

> > Stepping through the values by 2's ?
>
> ...is effectively by 3's (why?) - memory access error at a=171.

To see if processing less data got you through.


> > Then everything that goes to the debug stream will go to the file as well.
> > http://www.povray.org/documentation/view/3.6.1/222/
>
> This produced just a seemingly endless rush of vertex numbers (from 0 to
> 999999), without any hint at the reason for the memory access error.

Yeah, well - whatever you send to the debug stream gets written there, and
whatever you don't - doesn't.

I edited your file for (my) clarity, added some time stamps, and ran it.
Near the end of your file you have a missing closing parentheses:

#write (ES  "  finish{ F_Earthslice }\n")
#write (ES, "    }\n")   // <--------- missing closing )

Once I fixed that, it ran just "fine".
It's slow as hell and uses a LOT of memory, which is likely your problem.  You
probably run OUT of memory somewhere.

-------------------- STARTING NESTED LOOP 1 --------------------
NESTED LOOP 1 Time elapsed = 62.536 sec.
-------------------- STARTING NESTED LOOP 2 --------------------
NESTED LOOP 2 Time elapsed = 19.933 sec.
-------------------- STARTING NESTED LOOP 3 --------------------
NESTED LOOP 3 Time elapsed = 118.422 sec.
-------------------- STARTING LOOP 1 --------------------
LOOP 1 Time elapsed = 20.327 sec.
-------------------- STARTING LOOP 2 --------------------
LOOP 2 Time elapsed = 18.262 sec.
-------------------- STARTING NESTED LOOP 4 --------------------
NESTED LOOP 4 Time elapsed = 33.183 sec.
-------------------- STARTING LOOP 3 --------------------
LOOP 3 Time elapsed = 47.177 sec.
Total Time elapsed = 319.840 sec.

It takes 5 minutes to process your one (not very large) image file.
Parse Time:       0 hours  5 minutes 20 seconds (320.167 seconds)

So, I would suggest that you take a hard look at your code, see what is NEEDED
and what is not, trim the fat, and then optimize your code to use less than 7+
loops to construct a mesh from the pixel data.

You should be able to skip over performing certain operations using #if (value =
0) or #if (value = finalvalue-1) or #if (value = finalvalue)

That 3rd nested loop is an absolute killer.
You do a lot of the same calculations many MANY times.
Pre-calculate equation terms before all of your conditional statements, and then
you not only save time but also make the expressions much shorter and easily
read.

Don't tackle such large data sets until you have the basic code at least
somewhat optimized.   Crop out a small portion of the original image as a test
area, or use a larger step size to skip over some percentage of the data.

You're processing over a million tokens in theis scene with that image, so you
likely just don't have the system resources to handle the array data that you're
piling up.   If you can just use temporary variable that can get redeclared in
the next step of the loop, you might be just fine.

Post a reply to this message

Also, for nested loop 3, you're using the eval_pigment MACRO from an include
file "functions.inc"

All that does is:
#macro eval_pigment(pigm, vec)
    #local fn = function { pigment { pigm } }
    #local result = (fn(vec.x, vec.y, vec.z));
    result
#end

Somehow this magically returns a vector quantity, rather than a scalar.  :O

But the point is, that maybe declaring the function at the beginning of the
scene and just evaluating it when you need to without a lot of intermediate
steps and storage variables might cut down on memory usage and speed things up a
bit.

Post a reply to this message

Hi(gh)!

On 02.08.20 21:54, Bald Eagle wrote:

> I edited your file for (my) clarity, added some time stamps, and ran it.
> Near the end of your file you have a missing closing parentheses:
> 
> #write (ES  "  finish{ F_Earthslice }\n")
> #write (ES, "    }\n")   // <--------- missing closing )

Okay, that's the ticket! Thank you VERY much... that's a whole asteroid 
off my heart! What a stupid little mistake...

Probably the seemingly endless #write sequence led to memory overload...

> 
> Once I fixed that, it ran just "fine".
> It's slow as hell and uses a LOT of memory, which is likely your problem.  You
> probably run OUT of memory somewhere.

The original version by Melody was even much, much slower - generating a 
3601 by 3601 mesh took some cool 200 days! My version does it in two and 
a half hours...

> That 3rd nested loop is an absolute killer.
> You do a lot of the same calculations many MANY times.
> Pre-calculate equation terms before all of your conditional statements, and then
> you not only save time but also make the expressions much shorter and easily
> read.

I don't understand what you mean - the very formula by which the sums of 
the vector cross products are calculation depends on the position of the 
respective vertex, which are divided into nine different cases: lower 
left, lower right, upper left and upper right corners, lower, left, 
right and upper side, center. I cannot pre-calculate anything! At least 
not without the risk of messing up the code completely... never change a 
running system!

> 
> Don't tackle such large data sets until you have the basic code at least
> somewhat optimized.   Crop out a small portion of the original image as a test
> area, or use a larger step size to skip over some percentage of the data.
> 
> You're processing over a million tokens in theis scene with that image, so you
> likely just don't have the system resources to handle the array data that you're
> piling up.   If you can just use temporary variable that can get redeclared in
> the next step of the loop, you might be just fine.
> 

Before that parenthesis went lost, it worked fine... probably slower 
than necessary, but it worked!

See you in Khyberspace!

Yadgar

Post a reply to this message

> Okay, that's the ticket! Thank you VERY much... that's a whole asteroid
> off my heart! What a stupid little mistake...
>
> Probably the seemingly endless #write sequence led to memory overload...

Well - it's those damned pesky little parentheses again.


> The original version by Melody was even much, much slower - generating a
> 3601 by 3601 mesh took some cool 200 days! My version does it in two and
> a half hours...

OMG - that's ridiculous.
Use what you have, but start experimenting with making a version that's MUCH
faster.


> I don't understand what you mean

Just casually look at that block of code.
How many individual times do you calculate "xdim+(b-1)"?
Using that as a starting point, how many arithmetic operations can you "roll
into one"?

I cannot pre-calculate anything!
Yes you can.  Look, think, TRY.
Sometimes small things, sometime larger patterns reveal themselves.


> Before that parenthesis went lost, it worked fine... probably slower
> than necessary, but it worked!

Dude - if you know you can use a table saw, why rip an 8ft board by hand?
I'm using a 21600 x 10800 pixel heightfield for an _isosurface_, and I _render_
that in minutes, not hours....
How are you going to process _multiple_ tiles?   :O

Post a reply to this message

Le 2020-08-02 à 06:54, Jörg "Yadgar" Bleimann a écrit :
> Hi(gh)!
> 
> I now have run mesh2writer.pov in various RAM configurations including 
> only one 4 GB strip or one 8 GB strip in slot 1 - but always I got the 
> memory access error in the 22nd outer iteration (at 1000 by 1000 data 
> points; the number of the inner iteration varies wildly!) of the last 
> nested loop:
> 
> #if (!(tilename="n80e014" | tilename="n80e016" | tilename="n80e017"))
>    #declare i=0;
>    #write (ES, "  texture_list")
>    #write (ES, "  {\n")
>    #write (ES, concat("    ",str(NumTextures, 1, 0),"\n"))
>    #declare a=0;
>    // #declare c=0;
>    #while (a<ydim)                    // outer loop
>      #declare b=0;
>      #while (b<xdim)                    // inner loop
>        #declare C_Texture = eval_pigment(P_Texture, 
> <(0.5+b)*(1/(xdim-1)), (0.5+a)*(1/(ydim-1)), 1>);
>        #warning concat("Iteration ", str(a, 1, 0), "-", str(b, 1, 0))
>        #write (ES, "    texture\n")
>        #write (ES, "    {\n")
>        #write (ES, "      pigment\n")
>        #write (ES, "      {\n")
>        #write (ES, concat("       color rgb <",vstr(3, C_Texture, ",", 
> 1, 7),">\n"))
>        #write (ES, "      }\n")
>        #write (ES  "     finish{ F_Earthslice }\n")
>        #write (ES, "    }\n"
>        // #declare c=c+1;
>        #declare b=b+1;
>      #end
>      #declare a=a+1;
>    #end
>    #write (ES, "  }\n")
> #end
> 
> // end of code
> 
> I just can't figure out what should be wrong with this code - or is it 
> just a bug in POV-Ray 3.7?
> 
> See you in Khyberspace!
> 
> Yadgar

Given that the error happens during a write operation to the disk, I 
suspect that the actual error may be related to disk access issue rather 
than a RAM issue.
What is the size of the file when the error happens and how much disk 
space is available?
How the current file size relate to your cluster size ?

Try creating a dummy file before the run. For example, save a small 
image of about 8 to 10K. Does that cause the timing of the error to change ?


Alain

Post a reply to this message