Hi there,

For the images I am making I need lots of tiny, simple objects: grass, hair,
particle systems can be simulated with these.
Unfortunately, if I have 500,000 spheres or more, main memory seams to be
completely full, and PovRay is constantly swapping during the render (and it
takes about 30 min to get to the first pixel - I didn't even wait for the
second). I got 256 mb of main memory, and I think 500,000 spheres should fit
in there. (of course there are other objects too in the scene, but then,
what's a sphere in bytes?)
I tried to use triangle meshes: these do use less memory, but they take much
longer to render, which is, essentially, the point.
Anyway my idea was to make something like a "sphere mesh": a thing that does
the same as a union of spheres, but with less memory used. I think I need
only 12*8 bytes per sphere (transformation matrix plus translation vector).
My question is: Do you think this is a good idea? If yes, why hasn't anyone
done it before? If not, well, why not?
Has anybody tried anything similar? Any advices? I would just have analyzed
what a union of spheres does (maybe with the debugger) and tried to do the
same with a different memory layout, and without virtual functions...

Niki

Post a reply to this message

On Wed, 28 Aug 2002 10:44:59 +0200, "Niki Estner" <nik### [at] freenetde>
wrote:
> I got 256 mb of main memory, and I think 500,000 spheres should fit
> in there.

I'm sure it fits there and the memory consumption is caused by the complicated
texturing, media, csg, unnecessary object copying or something. Posting source
could help in investigation.

> Anyway my idea was to make something like a "sphere mesh": a thing that does
> the same as a union of spheres, but with less memory used. I think I need
> only 12*8 bytes per sphere (transformation matrix plus translation vector).
> My question is: Do you think this is a good idea? If yes, why hasn't anyone
> done it before? If not, well, why not?

Becouse it would use no less memory than spheres. You can crate one "mesh
sphere" and clone it many times as it is done internally for copied meshes
however every mesh will store pointer to original structure + transformation
while original sphere store only four floats (center + radius) for not
transformed sphere. You have to store texture and other object modifiers the
same way in both 'versions' of sphere.

> Has anybody tried anything similar?

I have tried clone object for large csgs and will try again it probably for
3.5 sources.

ABX

Post a reply to this message

I guess this is the wrong newsgroup then, but if you would like to answer,
I'd be pleased anyway. Otherwise I'd post to the advanced-users group.

Posting the whole scene doesn't make much sense since it contains some
height fields, image maps... Too large to post.

But if I create another scene which contains not much more than:

union {

#declare i=0;

#while (i<500000)

sphere { <i*0.01,i*0.02,i*0.03>, 0.04 scale <0.5,0.6,0.7> rotate
<i,i*2,i*3> }

#declare i=i+1;

#end

pigment { colour Red }

}

memory is full as well. I didn't wait for the parser to finish, but with
100,000 spheres, povray needs about 100 MB of memory! (measured with the
physical memory indicator of the W2K task manager).

Is there something wrong with that code? I thought putting the spheres into
a union was a good idea, so they all share one pigment. Also, the texture is
quite simple, as you can see.

Niki

"Niki Estner" <nik### [at] freenetde> schrieb im Newsbeitrag
news:3d6c8f20@news.povray.org...
> Hi there,
>
> For the images I am making I need lots of tiny, simple objects: grass,
hair,
> particle systems can be simulated with these.
> Unfortunately, if I have 500,000 spheres or more, main memory seams to be
> completely full, and PovRay is constantly swapping during the render (and
it
> takes about 30 min to get to the first pixel - I didn't even wait for the
> second). I got 256 mb of main memory, and I think 500,000 spheres should
fit
> in there. (of course there are other objects too in the scene, but then,
> what's a sphere in bytes?)
> I tried to use triangle meshes: these do use less memory, but they take
much
> longer to render, which is, essentially, the point.
> Anyway my idea was to make something like a "sphere mesh": a thing that
does
> the same as a union of spheres, but with less memory used. I think I need
> only 12*8 bytes per sphere (transformation matrix plus translation
vector).
> My question is: Do you think this is a good idea? If yes, why hasn't
anyone
> done it before? If not, well, why not?
> Has anybody tried anything similar? Any advices? I would just have
analyzed
> what a union of spheres does (maybe with the debugger) and tried to do the
> same with a different memory layout, and without virtual functions...
>
> Niki
>
>

Post a reply to this message

On Thu, 29 Aug 2002 17:51:03 -0400, Niki Estner quoth:

> I guess this is the wrong newsgroup then, but if you would like to
> answer, I'd be pleased anyway. Otherwise I'd post to the advanced-users
> group.
> 
> But if I create another scene which contains not much more than:
> 
> union {
> #declare i=0;
> #while (i<500000)
> sphere { <i*0.01,i*0.02,i*0.03>, 0.04 scale <0.5,0.6,0.7> rotate
> <i,i*2,i*3> }
> #declare i=i+1;
> #end
> pigment { colour Red }
>
> memory is full as well. I didn't wait for the parser to finish, but with
> 100,000 spheres, povray needs about 100 MB of memory! (measured with the
> physical memory indicator of the W2K task manager).

This definitely needs looking into: a quick run of the above scene shows

>> Peak memory used:        359841866 bytes

or about 720 bytes per sphere.  If I texture the spheres individually and
remove the union{}, things get even worse:

>> Peak memory used:        621841342 bytes

-- 
Mark

Post a reply to this message

On Fri, 30 Aug 2002 01:21:32 -0400, Mark Wagner <mar### [at] gtenet> wrote:
> or about 720 bytes per sphere.

whole sphere structure in discussed example cause at least:
- 11 pointers
- 1 int
- 1 unsigned long
- 7 single float
- 34 double float

and it contains:
- pointer to sphere and pointers to local structures
- transformation
- center and radius
- bounding box vectors
- other parameters

I don't know how much of memory use vista buffer.
I don't know how much of memory use other structures required for rendering.

ABX

Post a reply to this message

Mark & Niki,
I am sure some of the developers better know the numbers, but if you are willing
to listen to a relative newbie, here is how I see it.

I believe Pov-Ray uses a 4 by 4 matrix of doubles internally for each transform
of anything.  This would account for 16 x 16 = 256 bytes of what you see for
each sphere.  Guessing a little, but automatic bounds,  the light buffers and
vista buffers for half a million objects has to be pushing 256 bytes per object
- this you could actually test by turning these options off. If you try it, I
would not bother completing the render as the performance will be awful.  Just
those things  get us to 512 bytes per sphere.  Then we have the sphere itself at
4 x 16 = 64 bytes.  Probably at least 4 pointers at 4 bytes is another 16 bytes
a sphere. Let see that gets us to 592 bytes per sphere. Just to store the
calculated image for say an 800 x 600 render (Internally 3 x 16 bytes a pixel)
is about 46 bytes per sphere and there is bound to be pointers from each of
those screen locations to something for another 4 bytes per sphere.  That gets
us to 642 bytes a sphere.  I myself cannot account for the remaining 80 bytes
per sphere, but I would make a bet there are people here who know.

Mark, I think the doubling you saw when you individually textured is simply
because instead of storing one texture for all spheres you must now  store
500,000 and these texture structures. Even when textures are very, basic they
are not small and each will most probably be themselves transformed which means
pointers at the very least.

I guess I don't see this amount of storage as being that out of line - if
performance and reasonable accuracy are you goals.  I expect it is possible to
more compactly represent this data, but likely only at some performance expense
and code complexity.  Someone, in one of the Pov-Ray news groups, mentioned the
idea of an option to use floats everywhere instead of doubles to reduce storage.
Pov-Ray would run this way, but I doubt you would like the inaccuracies which
would be apparent in all but the simplest images.

Memory is reasonably priced these days - perhaps Pov-Ray costs something after
all! :-)
Bill P.

Post a reply to this message

On Fri, 30 Aug 2002 03:29:33 -0400, "William F. Pokorny"
<pok### [at] attglobalnet> wrote:
> I believe Pov-Ray uses a 4 by 4 matrix of doubles internally for each transform
> of anything.

Every transformation structure contains 2 matrixes: normal and inversed one.

ABX

Post a reply to this message

This question may be dumb, but why does a sphere need a bounding box? It
isn't used during rendering, is it?
Also, why does an ellipsoid need a center and a readius? The light ray must
be transformed into "sphere space"
anyway, couldn't center and radius be done right with that, too
(scale+translate of unity sphere)? If I say all my spheres share one
texture, couldn't I save some of the pointers for texture, interior,...?
That was basically the idea when I started to think about my "sphere_mesh".
I began to implement it, it needs one transformation per sphere (as already
said, 2x4x4x8 bytes). Unfortunately, my spheres don't use e
vista/light-buffer anymore of course, which makes them quite slow. I read
(in the vlbuffer comments) that the triangle-mesh and blob objects have
their own "vista-buffering-like" optimizations. Both objects are too slow
for me, but I'll take a deep look at the mesh-bounding-trees, I guess.
Now, finally, the last dumb question: All matrices are 4x4, while only 4x3
is needed (3x3 transformation matrix plus 3x1 translation vector). Couldn't
I store the matrices in 4x3 format, "unpacking" them when needed (64 bytes
per sphere)? Also, I could try to store the matrix with floats instead of
doubles (another 96 bytes).

All this doesn't make nice code, and it's no fun doing it anyway, so if
anyone got any better ideas for me, please tell me so.

Niki

"ABX" <abx### [at] abxartpl> schrieb im Newsbeitrag
news:uf3umuo52v40rumoc9smuvfnhr3lfim85r@4ax.com...
> On Fri, 30 Aug 2002 01:21:32 -0400, Mark Wagner <mar### [at] gtenet>
wrote:
> > or about 720 bytes per sphere.
>
> whole sphere structure in discussed example cause at least:
> - 11 pointers
> - 1 int
> - 1 unsigned long
> - 7 single float
> - 34 double float
>
> and it contains:
> - pointer to sphere and pointers to local structures
> - transformation
> - center and radius
> - bounding box vectors
> - other parameters
>
> I don't know how much of memory use vista buffer.
> I don't know how much of memory use other structures required for
rendering.
>
> ABX

Post a reply to this message

On Fri, 30 Aug 2002 10:10:24 +0200, "Niki Estner" <nik### [at] freenetde>
wrote:
> This question may be dumb, but why does a sphere need a bounding box? It
> isn't used during rendering, is it?

Bounding boxes usually are not used in theirs parent objects. They are for
vista buffer for example.

> Also, why does an ellipsoid need a center and a readius? The light ray must
> be transformed into "sphere space"
> anyway, couldn't center and radius be done right with that, too
> (scale+translate of unity sphere)?

It would be much slower.

> If I say all my spheres share one
> texture, couldn't I save some of the pointers for texture, interior,...?

IIRC it is done similiar way, but I can be wrong

> All this doesn't make nice code, and it's no fun doing it anyway, so if
> anyone got any better ideas for me, please tell me so.

Buy RAM ? :-)

ABX

Post a reply to this message

> > [using a matrix transformation instead of center/radius]
> It would be much slower.
For a sphere, I understand this, but for an ellipsoid? The ray has to be
multiplied with a matrix anyway, the center/radius scale/translate matrix
could be combined with the matrix that's already there (that made an
ellipsoid out of the sphere). Wouldn't that save a vector-subtraction (no
big time saving, but not slower at least)

> IIRC it is done similiar way, but I can be wrong
What's IIRC?

> Buy RAM ? :-)
Guess I'll gonna do that...
But admit it: when you read my problem first, you also thought: 500,000
spheres - no problem with 256MB.

Post a reply to this message