Hi(gh)!

A second "Georgian" elevation data tile has been processed into a mesh2: 
n42e044 (the naming is according to the coordinates of the southwestern 
corner). The terrain texture is a 240 by 240 pixel section from the July 
2004 version of the global Blue Marble Mosaic (86400 by 43200 pixels)

First, I started with a overall view from south and 40,000 metres above 
sea level... somewhere at the eastern (right) edge of the central 
glaciated area must be Mt. Kazbek (5034 metres), which should be quite 
remarkable, when moving the camera closer (as I plan for today, trying 
to capture the Kazbek massif from Stepantsminda).

Contrary to Bald Eagle's recommendations, precalculations and 
concatenations of #write operations did not help much in increasing 
parsing speed of the mesh2 generating script - calculating a mesh2 from 
the n42e044 heightfield still took 2 hours, 32 minutes. I need a faster 
computer! 16-core Ryzen Threadripper fitted with 128 GiB of RAM or 
something like that...

I also will try today whether my 24-GiB hexacore is able to render 
scenes containing two instead of just one mesh2 terrain relief tile, 
adding the Caucasus as backdrop for Tbilisi... back in April, this 
worked quite well with Svalbard, but these reliefs were mostly flat sea 
floor.

See you in Khyberspace!

Yadgar

Post a reply to this message

Attachments:
Download '2020-08-15 greater caucasus (n42e044) from south and 40,000 metres.jpg' (95 KB)

Preview of image '2020-08-15 greater caucasus (n42e044) from south and 40,000 metres.jpg'

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> ...
> Contrary to Bald Eagle's recommendations, precalculations and
> concatenations of #write operations did not help much in increasing
> parsing speed of the mesh2 generating script - calculating a mesh2 from
> the n42e044 heightfield still took 2 hours, 32 minutes. I need a faster
> computer! 16-core Ryzen Threadripper fitted with 128 GiB of RAM or
> something like that...

while I can see that with 2.5 hours parse time you'd not want to run code two or
three times just to compare, had hoped for a more precise measure than "did not
help much".  a big UK supermarket has the advertising slogan "every little
helps"  :-)  (I'm guessing that there could be other opportunities to improve
that code, in addition to the loop you've shown)

I'll have most of Tuesday to myself, so if you want, post an archive with code
and image(s).  I'm curious, and would be happy to have a look.

(I think #cores and threads in a new computer, even if slightly higher clocked,
won't make much difference wrt single threaded parsing; more RAM might if swap
is used.  hopefully someone will correct me if this assumption is false)

> ...


regards, jr.

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> Contrary to Bald Eagle's recommendations, precalculations and
> concatenations of #write operations did not help much in increasing
> parsing speed of the mesh2 generating script - calculating a mesh2 from
> the n42e044 heightfield still took 2 hours, 32 minutes. I need a faster
> computer! 16-core Ryzen Threadripper fitted with 128 GiB of RAM or
> something like that...

generating a  3601 by 3601 mesh took some cool 200 days! My version does it in
two and a half hours...

Trying to help here with some constructive criticism.

Yadgar, I'm sorry, but something is _wrong_.
3601 x 3601 is only 12,967,201 pixels.
In its simplest terms, you're taking those pixels, and instead of x, y, RGB
you're writing some data to disk as x, y, (altitude).
That should take _minutes_.

I'm currently using a 21600 x 10800 image as a heightfield to sample my altitude
values and whether I'm rending that as an isosurface - which HAS to sample every
single pixel and then do all the isosurface geometry and the render the WHOLE
PLANET,

http://news.povray.org/povray.binaries.images/thread/%3Cweb.5f0a6d9a6f747b6fb0b41570%40news.povray.org%3E/

or if I'm sampling the data and then writing 16 control points per bezier patch,
plus the rest of the data for [currently a mere] 1500 separate include files,

http://news.povray.org/povray.binaries.images/thread/%3Cweb.5f3095315d69181b1f9dae300%40news.povray.org%3E/

it doesn't take anywhere near that long.


In my last analysis, I think I counted 7 separate loops, at least one of those
was nested.  There is a forest of calculations that you do, which although may
not be the biggest processing bottleneck, is sure to be a debugging nightmare
later on down the road.

I would suggest that you go through you get a coffee, take a deep breath and
enter a state of deep relaxation, and promise yourself that you refuse to accept
ANY assumptions no matter how small.
Then write out a small simple GOAL oriented description of what the scene should
do.   Not HOW it should be done, but WHAT needs to be accomplished.

Then take a tiny 640x480 noise image - you can make it in 5 sec with the bozo
pattern - and using nothing but spheres map out the vertices that you need in a
small test scene.  When you have that working, try it with your tile and see how
long it takes.

Moving back to the 640 x 480, you then add some connecting cylinders to make a
wireframe.  Then run your tile.

Do this all from scratch.
You shouldn't need a lot of loops or complex hard-to-follow expressions.
If you have very similar command lines in your code, get all of the operators
and subexpressions to line up so that you can tell at a glance if something is
off.

#local thisVector = <x * 123.0/ 7 + oneValue, y *  11.2/25, z * 0.04/7>;
#local thatVector = <x *-286.5/24 + twoValue, y * 364.0/ 3, z * 1.00/7>;

If you have a complex expression that gets evaluated over and over again, then
declare that as a variable or write a macro or use a function.   It will make
the above expression shorter, easier to read, easier to debug, easier to spot
errors, and you won't have 3 lines or 5 lines, or 40 lines that you need to edit
if there's a change in a fundamental calculation.
Think about it like a spreadsheet.  ONE change should be able to affect 4000
cells.  You shouldn't have to edit the equations in 4000 cells because one value
changed.  And simplifying one complex expression that gets used 4000 times - or
_12,967,201 times_ HAS to help.  You know that this HAS to be true.

It doesn't make any difference HOW many resources you have if you're not using
them efficiently.  Instead of _wasting_ all of your time trying to debug code
that doesn't work (this early on in the project) (and will be a recurring
theme), _spend_ your time writing code that doesn't NEED to be debugged.
And if you do the same thing with the logic and flow of the code, it will surely
be faster and nimbler.

I'm simply not accepting that taking ONE single image and converting it into a
mesh2 object takes THAT long.   Surely there are software packages that could do
that with an entire directory of such images in a FAR shorter amount of time.
And POV-Ray SDL and your computer cannot possibly be THAT slow.

I was trying to stitch together bezier patches, and overall that took me YEARS
to learn how to do properly.   deCasteljau was a _terrible_ way to go about
that, and it was only after TOK suggested that I use Bernstein polynomials that
things got set right.   It still took me MONTHS, with TOK's patient guidance and
clipka's merciless percussive disciplining and divine intervention for me to
work everything out and get the math and the code straight.

And that was just for a teensy tiny little torus.   It was painful.   It was a
daily struggle.   But I rewrote a LOT of "working" code and I _learned_ a lot.

So maybe just try to take a step back, experiment with different methods,
simpler model scenes, and alternative ways to assemble the data, and you may
have an epiphany or two about what you actually need to do and what is going on
in your code.   And you will be MUCH happier in the long run.

Post a reply to this message

Hi(gh)!

On 15.08.20 15:19, Bald Eagle wrote:
> =?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> 
>> Contrary to Bald Eagle's recommendations, precalculations and
>> concatenations of #write operations did not help much in increasing
>> parsing speed of the mesh2 generating script - calculating a mesh2 from
>> the n42e044 heightfield still took 2 hours, 32 minutes. I need a faster
>> computer! 16-core Ryzen Threadripper fitted with 128 GiB of RAM or
>> something like that...
> 
> generating a  3601 by 3601 mesh took some cool 200 days! My version does it in
> two and a half hours...
> 
> Trying to help here with some constructive criticism.
> 
> Yadgar, I'm sorry, but something is _wrong_.
> 3601 x 3601 is only 12,967,201 pixels.
> In its simplest terms, you're taking those pixels, and instead of x, y, RGB
> you're writing some data to disk as x, y, (altitude).
> That should take _minutes_.

I can't write binary data to disk as I have to build a mesh2 object - 
AFAIK this can only be achieved by writing ASCII text!

See you in Khyberspace!

Yadgar

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> I can't write binary data to disk as I have to build a mesh2 object -
> AFAIK this can only be achieved by writing ASCII text!

No one said anything about writing binary data.
(but yes you can with POV-Ray)

Now, tell the truth.  You and Sven are brothers, or at least cousins.

http://news.povray.org/povray.general/thread/%3C59799a41%241%40news.povray.org%3E/?mtop=417481

Perhaps there is a feature of MeshLab or PoseRay or an STL-file software,
perhaps Blender - that can take an image map and convert it into a mesh.
There are some cloud computing solutions out there - maybe look into those.

Perhaps Maurice already has some sort of tool in Blender that can easily handle
this.

The next question is: do you really need this kind of extreme resolution?
At this point?

Can you simply use a heightfield, or a smaller image, or step through the image
every nth pixel?

I just looked at your old mesh file, and it's 24+MB.   That's a ton of
information for a single tile.

Post a reply to this message

Hi(gh)!

On 16.08.20 01:22, Bald Eagle wrote:
> =?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> 
>> I can't write binary data to disk as I have to build a mesh2 object -
>> AFAIK this can only be achieved by writing ASCII text!
> 
> No one said anything about writing binary data.

Then I don't understand you at all... must be the August heat in here!

> (but yes you can with POV-Ray)

I know... but it won't help me! And don't forget: it's not just a basic 
mesh2, it's a mesh2 with smooth triangles and per-vertex textures! It 
has to have smooth triangles because otherwise there would be too much 
artifacts (gaps in the surface) - even now I sometimes find tiny white 
dots, but at least anti-aliasing is able to hide this.

> Now, tell the truth.  You and Sven are brothers, or at least cousins.

No, we are not.

> Perhaps there is a feature of MeshLab or PoseRay or an STL-file software,
> perhaps Blender - that can take an image map and convert it into a mesh.
> There are some cloud computing solutions out there - maybe look into those.
> 
> Perhaps Maurice already has some sort of tool in Blender that can easily handle
> this.

Who is Maurice?

> 
> The next question is: do you really need this kind of extreme resolution?
> At this point?

Hmmm, that's a question not too easy to answer... my aim in the long run 
is a full-fledged open sources Earth model with realistic detail in 
"pedestrian" perspective - POVEarth. If I just would produce animations 
of overflights at 10,000 metres above sea level (e. g. for a new version 
of my "Earth Flight" project - see 
https://www.youtube.com/watch?v=u4fCO2ccm-w ) then probably 1800 by 1800 
or even 900 by 900 data points will do. But, as I mentioned in the 
"Georgia on my mind" thread, now I made contact with an avid Georgian 
POVer, who is eager to see some background relief for his modelings of 
historic Tbilisi - and in this case, 3601 by 3601 looks of course much 
better!

> Can you simply use a heightfield, or a smaller image, or step through the image
> every nth pixel?

Heightfields don't follow Earth's curvature, they are "flat", smaller 
images or larger steps I used with test renders, yes, they render pretty 
fast, but at a "pedestrian" perspective, they are much too crude.

> I just looked at your old mesh file, and it's 24+MB.   That's a ton of
> information for a single tile.

My full-resolution mesh2 files are a cool 3 GiB! And it's obvious that, 
at 24 GiB RAM, only two of them already cause POV-Ray to crash... I need 
MUCH more RAM and a MUCH faster CPU! And a private little dual fluid 
nuclear reactor on my balcony to power this beast of a computer...

See you in Khyberspace!

Yadgar

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> > No one said anything about writing binary data.
>
> Then I don't understand you at all... must be the August heat in here!

> it's not just a basic
> mesh2, it's a mesh2 with smooth triangles and per-vertex textures! It
> has to have smooth triangles because otherwise there would be too much
> artifacts (gaps in the surface) - even now I sometimes find tiny white
> dots, but at least anti-aliasing is able to hide this.


> Who is Maurice?

Maurice Raybaud - "Mr", who writes / maintains the Blender-to-POV-Ray exporter.

> > The next question is: do you really need this kind of extreme resolution?
> > At this point?
>
> Hmmm, that's a question not too easy to answer... my aim in the long run
> is a full-fledged open sources Earth model with realistic detail in
> "pedestrian" perspective - POVEarth.


> Heightfields don't follow Earth's curvature, they are "flat", smaller
> images or larger steps I used with test renders, yes, they render pretty
> fast, but at a "pedestrian" perspective, they are much too crude.
>
> > I just looked at your old mesh file, and it's 24+MB.   That's a ton of
> > information for a single tile.
>
> My full-resolution mesh2 files are a cool 3 GiB! And it's obvious that,
> at 24 GiB RAM, only two of them already cause POV-Ray to crash...


Okay - you are biting off way too much using too few resources.
You might as well work out a Fourier transform with a handheld calculator.

I'm still convinced that your approach is far too excessively detailed for what
you actually need, and the method of generation is flawed, and therefore WAY too
slow.

I just created an animation of my whole planet in 46 min.
 360 frames of 24x24x6 tiles, each with 2 include files totaling 228 lines =
3456 x 228 lines = 787,968+ lines of code for each frame
x 360 frames = 283,668,480 lines
That's generating all 3456 tiles from scratch, with an index, reading both
include files in, and _rendering the scene_ 360 times - in 46 minutes.

your 3601x3601 has 16x the number of data points than I have lines, but I'mm
processing 22.5 times more ASCII information than you are in 0.375 of the amount
of time. or 2.66 times as fast.

That's like 60x as fast as whatever your script is doing.

Wouldn't you rather process that tile in ... 2 *minutes* ?

I'd like to see you create the renders that you want to make.
But you can't do it at the pace you're going, and nearly blowing up the computer
in the process.

Post a reply to this message

I came across this serendipitously [as always]

This is now 17 years ago.   :O

https://www.triplespark.net/render/img/earth/

More food for thought.

Post a reply to this message

Hi(gh)!

On 23.08.20 15:46, Bald Eagle wrote:
> I came across this serendipitously [as always]
> 
> This is now 17 years ago.   :O
> 
> https://www.triplespark.net/render/img/earth/
> 
> More food for thought.


The resolution of my ASTER data tiles is 60 times higher, i. e. one 
arcsecond per measuring point rather than one arcminute!

See you in Khyberspace!

Yadgar

Post a reply to this message

Yadgar has returned Bald Eagle's serve, and BE hammers the ball back into Y's
court....


Though in 17 years, we've had Moore's Law operating, other increases in speed,
due to more efficient data busses, HDD indexing and r/w times, information
density, and improvements in POV-Ray's fundamental operation (v 3.8).

No excuses.   Edit your code.   Eliminate unnecessary data!  <crack!>
Davros mandates syntactical Darwinism.
Find inefficiencies and eliminate them!   EXTERMINATE!  EXTERMINATE!

:D

Post a reply to this message