Hi(gh)!

Finally, I managed to re-code Melody's smooth triangle macro in a way 
that avoids the abysmally slow double nested loop (details and code see 
p.t.s-f) - now generating a 3601 by 3601 mesh2 takes 2 hours instead of 
200 days!

The origin of the elevation data is the Advanced Spaceborne Thermal 
Emission and Reflection Radiometer (ASTER) aboard the US-Japanese 
"Terra" satellite, which conducted an Earth surveying missions in 2009. 
The basic link is https://asterweb.jpl.nasa.gov/gdem.asp, however, the 
set-up of the data archive has changed much since I downloaded the data 
ten years ago. Meanwhile, an improved dataset (less artifacts) has been 
published, but I have no clue how to access it...

Back then, elevation data tiles came as GeoTIFF files, each one covered 
an area of 1 square degree at a resolution of 3601 by 3601 measuring 
points counted from south to north and from west to east. The 
northernmost row and the easternmost column of values are identical to 
the first row and column of the respective adjacent tile.

As this is the first version of the dataset, some gaps were still 
present; the 3DEM visualization software (download: 
https://freegeographytools.com/3dem_setup.exe) comes with an 
interpolation tool to fill these gaps. Be warned: only with small gaps 
up to a few dozens of missing points this works really satisfactory, 
larger gaps will result in artifacts like north-east trending terrain 
structures. Probably I'll still have to download the newer dataset in 
the future...

Using the 3DEM software, I generated ASCII data matrices from them. 
These text files contain elevation data rounded to 1 metre (in the very 
most cases), with two digits accuracy (which is mostly just .00), each 
separated by (mostly) one blank space.

To get these files readable with POV-Ray, I had to replace all these 
decimal points, zeroes and black spaces by a single comma. As a Linux 
user, I did this using a sequence of several sed commands:

swcorner=$1
sed 's/.[0-9][0-9] /,/g' $swcorner".txt" > $swcorner"_pov.txt"
sed 's/ //g' $swcorner"_pov.txt" > $swcorner".txt"
sed 's/,,/,/g' $swcorner".txt" > $swcorner"_pov.txt"
mv -f $swcorner"_pov.txt" $swcorner".txt"

with $swcorner being the geographic coordinates of the south-western 
corner of each ASCII data tile, such as n80e014, being also the file 
name of the tile.

 From these ASCII tiles processed for use with POV-Ray, I generated 
red/green 16bit heightfields using a POV-Ray script (see p.t.s-f). 
Originally, I did the conversion directly from ASCII text matrix to 
mesh2 object, but I as also want to produce mesh2s from joined and cut 
ASTER tiles, I resorted to this graphic step inbetween.

The next step is to generate the mesh2 object, readily with smoothed 
triangles (also see details on p.t.s-f), which results in huge .inc 
files of 1.5 to 3 gigabytes.

Finally, the scene proper is rendered with my standard "POVEarth" script 
from 2013. Textures are gleaned from the highest-resolution NASA Blue 
Marble mosaics (July 2004 version), each one 240 by 240 pixels (in the 
future, this may change, as nowadays LANDSAT tiles at 1200 by 1200 
pixels per square degree are available), however, as Blue Marble does 
not cover areas beyond 80°N properly, I have to improvise.

Much later on, there may more detailed bitmap textures even for these 
high Arctic regions derived from ground photos...

For the time being, neither bathymetry (the ocean is just an uniform 
basin some 500 metres deep) nor sea ice cover is taken into account - 
this also may change in the future.

Also at the moment, the sky is a simple blue sky_sphere; real atmosphere 
and night sky will be added in the not-too-distant future, later on also 
cloud cover derived from real current data.

I now start rendering all 8000+ terrain tiles and presenting them here, 
tile by tile, simultaneously, I'll set up a new 2-terabyte webspace 
under the domain www.povearth.de to make the mesh2 "earth slices" and 
textures available to the international POV-Ray community - as I want to 
see POVEarth GROW and FLOURISH! I want to see you populating these 
barren terrain slices with vegetation, rivers, roads and buildings, also 
people, animals, vehicles, just literally everything that belongs to 
real Earth!

After this intial stage of generating and rendering terrain slices in 
the first place is through, I'll focus on detailing my favourite places 
on Earth - Afghanistan (i. e. POVghanistan) and Cologne.

A later development also could be a real-time interactive version of 
POVEarth, programmed in X3D...

###

Here now my first rendered terrain tile: Moffen Island, a tiny speck of 
land north of Spitsbergen (n80e014), due to its lowness, it is not 
glaciated, but contains a huge lagoon which makes it look somewhat like 
an Arctic atoll. The lagoon with its inner island was not really visible 
in the original ASTER data, so I had to manually cut it out after a 
schematic map of Moffen Island on Wikipedia.

More on Moffen Island: https://en.wikipedia.org/wiki/Moffen, see also 
the Wikimedia Commons entries for photos.

The first image attached here shows Moffen in an overhead view from 17 
kms altitude; the second one shows the island in an oblique view from 
south, at 1.7 kms above the sea; the third one is the same, but with a 
"sailor's view" only 5 metres above sea level; in the fourth one, the 
camera is placed at "pedestrian height" (1.7 metres) on the eastern rim 
of the lagoon, looking west; the fifth one is a view from the northern 
of two hills on the island to the southern one, also at "pedestrian height".

Each rendering took about 5 minutes on an AMD FX-6200 six-core machine 
with 24 GiB of RAM.

Any suggestions for improvements are welcome!

...er, any walrus model ready?

See you in Khyberspace!

Yadgar

Post a reply to this message

Attachments:
Download '2020-04-02 povearth, svalbard, moffen island, take 21 - view from 17 km above ground.jpg' (18 KB) Download '2020-04-02 povearth, svalbard, moffen island, take 22 - view from south, 1.7 km above ground.jpg' (101 KB) Download '2020-04-02 povearth, svalbard, moffen island, take 23 - 'sailor's view' from south, 5 metres above ground.jpg' (111 KB) Download '2020-04-02 povearth, svalbard, moffen island, take 24 - view from east across lagoon, 1.7 metres above ground.jpg' (88 KB) Download '2020-04-02 povearth, svalbard, moffen island, take 25 - view from northern hill to southern hill, 1.7 metres above groun' (17 KB)

Preview of image '2020-04-02 povearth, svalbard, moffen island, take 21 - view from 17 km above ground.jpg'

Preview of image '2020-04-02 povearth, svalbard, moffen island, take 22 - view from south, 1.7 km above ground.jpg'

Preview of image '2020-04-02 povearth, svalbard, moffen island, take 23 - 'sailor's view' from south, 5 metres above ground.jpg'

Preview of image '2020-04-02 povearth, svalbard, moffen island, take 24 - view from east across lagoon, 1.7 metres above ground.jpg'

Preview of image '2020-04-02 povearth, svalbard, moffen island, take 25 - view from northern hill to southern hill, 1.7 metres above groun'

Hi(gh)!

On 02.04.20 03:22, Jörg "Yadgar" Bleimann wrote:

> The origin of the elevation data is the Advanced Spaceborne Thermal 
> Emission and Reflection Radiometer (ASTER)
abhttps://www.spiegel.de/wissenschaft/medizin/coronavirus-so-erging-es-den-ersten-50-covid-19-patienten-in-der-uniklinik-aachen-a-7f9f2187-b925-4f87-b1da-9ce2c72494a1#kommentareoard
the US-Japanese 
> "Terra" satellite, which conducted an Earth surveying missions in 2009. 
> The basic link is https://asterweb.jpl.nasa.gov/gdem.asp, however, the 
> set-up of the data archive has changed much since I downloaded the data 
> ten years ago. Meanwhile, an improved dataset (less artifacts) has been 
> published, but I have no clue how to access it...

Available now at 
https://search.earthdata.nasa.gov/search/granules?p=C1546314043-LPDAAC_ECS&pg[0][id]=NASADEM_HGT_n37e064.zip&q=nasadem&m=48.61688194818677!8.12109375!6!1!0!0%2C2&tl=1570136741!4!!

Registration required!

Unless I have access to a real high-performance computer (not that 
shoddy consumer stuff), I probably will not generate mesh2 terrain 
slices from all available ASTER data tiles, as this would take me no 
less than 22 years - if I generated one mesh2 per day. Anyone who would 
like to contribute to the titanic task of POVEarth by downloading tiles 
and processing them is hereby invited to join me! During the coming 
weeks, I will set up a new webspace under the domain povearth.de where 
you may upload your terrain slices!

See you in Khyberspace!

Yadgar

See you in Khyberspace!

Yadgar

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> ...
> Registration required!
>
> Unless I have access to a real high-performance computer (not that
> shoddy consumer stuff), I probably will not generate mesh2 terrain
> slices from all available ASTER data tiles, as this would take me no
> less than 22 years - if I generated one mesh2 per day. Anyone who would
> like to contribute to the titanic task of POVEarth by downloading tiles
> and processing them is hereby invited to join me! During the coming
> weeks, I will set up a new webspace under the domain povearth.de where
> you may upload your terrain slices!

I'd be willing to process the odd slice/tile data, _but_, since I have no
interest in the data itself (only in "community support" :-)), I would not want
to register etc.  my suggestion is that you'll organise the whole thing a little
like SETI or Folding@Home, that is, I'd go to your website and find an archive
with the necessary POV-Ray, sed, etc scripts, and a form where I'd enter (with
email) a request for one slice/tile, and receive, by return, either the data
attached or, preferably, a link so I can 'wget' the file.  added advantage would
be no inadvertent duplication of work.


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:

> I'd go to your website and find an archive
> with the necessary POV-Ray, sed, etc scripts, and a form where I'd enter (with
> email) a request for one slice/tile, and receive, by return, either the data
> attached or, preferably, a link so I can 'wget' the file.  added advantage would
> be no inadvertent duplication of work.


This sounds good.
Even just a short how-to guide about downloading specific tiles and rendering
them would be a helpful jumpstart.

I haven't looked over the POV-Earth scene, but might I suggest that all of tiles
get systematically named and put in a subdirectory, and the pov file then checks
for the presence of a tile, and if not found, renders a default texture.

Post a reply to this message

Hi(gh)!

On 04.04.20 11:26, jr wrote:

> I'd be willing to process the odd slice/tile data, _but_, since I have no
> interest in the data itself (only in "community support" :-)), I would not want
> to register etc.  my suggestion is that you'll organise the whole thing a little
> like SETI or Folding@Home, that is, I'd go to your website and find an archive
> with the necessary POV-Ray, sed, etc scripts, and a form where I'd enter (with
> email) a request for one slice/tile, and receive, by return, either the data
> attached or, preferably, a link so I can 'wget' the file.  added advantage would
> be no inadvertent duplication of work.

Did I understand you correctly so far that you want me to set up the 
https://gnadlib.wordpress.com/2020/04/05/fahrrad/webspace in a way that 
POV-Ray immediately can access the original data files on the server 
(remote file access in POV-Ray? Must be a new feature in 3.7 which 
escaped my attention until now...) and process them?

This probably would not be possible, as 2 TB additional webspace is at 
the upper limit I can afford - and the original files come as *.zip from 
the NASA server...

See you in Khyberspace!

Yadgar

Now playing: Focus III - Answers? Questions! Questions? Answers! (Focus)

Post a reply to this message

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:

> Did I understand you correctly so far that you want me to set up the
> https://gnadlib.wordpress.com/2020/04/05/fahrrad/webspace in a way that
> POV-Ray immediately can access the original data files on the server
> (remote file access in POV-Ray?

It seems to me you did not.
If you provide a small toolkit of scripts that you're using to process the raw
data files, and links to the datafiles as well, then it would be a lot easier
for folks to get non-overlapping tile data, and quickly and easily process them.

If there's a way to remove or gray-out any links to tile files that have already
been downloaded, then that would ensure non-duplicate efforts.

> Must be a new feature in 3.7 which
> escaped my attention until now...) and process them?

Nope, you read too much into jr's request.
And you really should be planning a move to 3.8...   ;)  :)

I mean, I'm all signed up with an[other] account, but it's a bit confusing as to
how to find a specific square to render...

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> On 04.04.20 11:26, jr wrote:
> > ...
> Did I understand you correctly so far that you want me to set up the
> https://gnadlib.wordpress.com/2020/04/05/fahrrad/webspace in a way that
> POV-Ray immediately can access the original data files on the server
> (remote file access in POV-Ray? Must be a new feature in 3.7 which
> escaped my attention until now...) and process them?

no.  as Bald Eagle pointed out, you misunderstood (completely).

the website I referred to is "povearth.de".  and Bald Eagle's reply provides a
fine summary of what would be expected.


> This probably would not be possible, as 2 TB additional webspace is at
> the upper limit I can afford - and the original files come as *.zip from
> the NASA server...

in an earlier post you wrote "as a Linux user".  that gives you the opportunity
to use (significantly) better compression for the data than zip achieves; I'd
"re-package" the zipped data using either 'gzip(1)' or, better, 'xz(1)'.  (for
instance, I have a VRML ASCII data file which, using xz, crunched from just over
1G to just under 10M)

also, the 'sed' command features an '-i' option switch which allows you to
update the file, no need to make intermediate copies; you can supply multiple
expressions ('-e'), too.


regards, jr.

Post a reply to this message

Hi(gh)!

On 06.04.20 10:25, jr wrote:

> the website I referred to is "povearth.de".  and Bald Eagle's reply provides a
> fine summary of what would be expected.
> 
> 
>> This probably would not be possible, as 2 TB additional webspace is at
>> the upper limit I can afford - and the original files come as *.zip from
>> the NASA server...
> 
> in an earlier post you wrote "as a Linux user".  that gives you the opportunity
> to use (significantly) better compression for the data than zip achieves; I'd
> "re-package" the zipped data using either 'gzip(1)' or, better, 'xz(1)'.  (for
> instance, I have a VRML ASCII data file which, using xz, crunched from just over
> 1G to just under 10M)

I tried xz... but my mesh2 inc files are still about 80 MB after 
compressing - so I will still need some 700 GB of webspace make them 
available for download. I recently did some research in webspace prices 
- and found out that 2 TB would be MUCH too expensive for me, as they 
usually are around 500 US$/year... the largest webspace offer I could 
afford is 500 GB for 36 euros/year, and that probably would be not 
enough for all 8000+ tiles processed into compressed mesh2 inc files.

Most likely, I would store the POVEarth terrain data online as PNG 
heightfields - they are smaller than 10 MB per tile down the line.

However, I cancelled my "rock-o-data" webspace (hosted by the very same 
company which offers that bargain 500 GB), which will expire in November 
2021... after that, I'll migrate the POVEarth data to a new webspace 
under povearth.de!

> also, the 'sed' command features an '-i' option switch which allows you to
> update the file, no need to make intermediate copies; you can supply multiple
> expressions ('-e'), too.

Did not try out yet; first, I'll have to (re-)calculate the mesh2 from 
the n80e017 data tile, then render a few views and also trying to do a 
first multi-tile scene (from n80e016 and n80e017 tiles)... and finally 
get some hours of sleep! ;-)

See you in Khyberspace!

Yadgar

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> Hi(gh)!
>
> ...
> See you in Khyberspace!

just came across your .. brother-in-arms.  ;-)

<https://www.youtube.com/watch?v=8_bW3ab8YAk>


regards, jr.

Post a reply to this message

hi,

=?UTF-8?Q?J=c3=b6rg_=22Yadgar=22_Bleimann?= <yaz### [at] gmxde> wrote:
> ...
> >> This probably would not be possible, as 2 TB additional webspace is at
> >> the upper limit I can afford - and the original files come as *.zip from
> >> the NASA server...
> ...
> I tried xz... but my mesh2 inc files are still about 80 MB after
> compressing - so I will still need some 700 GB of webspace make them
> available for download. I recently did some research in webspace prices
> - and found out that 2 TB would be MUCH too expensive for me, as they
> usually are around 500 US$/year... the largest webspace offer I could
> afford is 500 GB for 36 euros/year, and that probably would be not
> enough for all 8000+ tiles processed into compressed mesh2 inc files.

on the source (zip) files -- you wrote a 3601x3601 tile needs ~1.5G.  that is
~115 bytes per pixel; likely, some/much of that information will not be
required.  suggest recoding those, to a custom file format.

on the mesh2 storage -- what if meshes were only created on-the-fly[*], when
needed for rendering that part of povearth?  sure, performance hits everywhere,
but within budget.

another "budgeting" option would be to focus on a single continent, initially,
and only produce those meshes.  (maybe, when you retire, your pension will allow
you to buy more cloud-storage.  :-))

[*] ini file 'Pre_{Frame,Scene}_Command'.


regards, jr.

Post a reply to this message