The PovBench (http://www.haveland.com/povbench/) has been obsolete for
long time now (several years, I would say). The skyvase scene is not slow
enough to measure current fastest computers accurately, and besides, the
list is full of bogus entries. Even if the entry is genuine, two entries
with a rendering time of 3 seconds just doesn't tell which one is better or
if they are equal.
  Moreover, using one simple scene to benchmark POV-Ray is too restrictive:
It measures just a very small percentage of POV-Ray features (eg. it doesn't
measure heavy memory usage or parsing speed).

  Wouldn't it be time to make a renewed POV-Ray benchmark? A much better
benchmark up to the current computer speeds?
  This benchmark could have the following features:

  1. It has more than one scene file, each one benchmarking its own important
area in rendering, for example one for raw raytracing speed, one which takes
long to parse, one which uses lots of memory, etc.

  2. These scene files should take a reasonable amount of time in current
computers. This amount should be chosen so that it will not be in the order
of a couple of seconds in a few years. For example they could take about 10
minutes each in an 1.2GHz Athlon.

  3. The submission of entries should be controlled. Of course it's difficult
to see if someone is just making a bogus entry if the numbers are credible,
but there could be, for example, a "trusted entry system": Entries from
trusted sources could get a mark showing that it's a trusted entry. Entries
not having this mark may or may not be true (and this should be clearly
stated in the page).

-- 
#macro N(D,I)#if(I<6)cylinder{M()#local D[I]=div(D[I],104);M().5,2pigment{
rgb M()}}N(D,(D[I]>99?I:I+1))#end#end#macro M()<mod(D[I],13)-6,mod(div(D[I
],13),8)-3,10>#end blob{N(array[6]{11117333955,
7382340,3358,3900569407,970,4254934330},0)}//                     - Warp -

Post a reply to this message

Warp wrote:
> 
>   The PovBench (http://www.haveland.com/povbench/) has been obsolete for
> long time now (several years, I would say). The skyvase scene is not slow
> enough to measure current fastest computers accurately, and besides, the
> list is full of bogus entries. Even if the entry is genuine, two entries
> with a rendering time of 3 seconds just doesn't tell which one is better or
> if they are equal.
>   Moreover, using one simple scene to benchmark POV-Ray is too restrictive:
> It measures just a very small percentage of POV-Ray features (eg. it doesn't
> measure heavy memory usage or parsing speed).
> 
>   Wouldn't it be time to make a renewed POV-Ray benchmark? A much better
> benchmark up to the current computer speeds?
>   This benchmark could have the following features:
> 
>   1. It has more than one scene file, each one benchmarking its own important
> area in rendering, for example one for raw raytracing speed, one which takes
> long to parse, one which uses lots of memory, etc.

Then, you would have multiple metrics. IMNSHO, it's a bad thing, because
everyone will wants soon it's own test scene. 
You should rather stick to a single scene, which should nevertheless
be a complex-looking and lovely one.
Rendering an animation set of frame may be an option, but it might easily
open the pandora box with the #if/#switch statements.
Anyway, a full INI file should be provided with the scene,
and every possible setting should be explicitely done in that INI file.

BTW, the rendered size should be huge (minimum of 1600x1200), with AA, 
and the scene should not be biazed toward any colour.
(Skyvase is too much blue !)

> 
>   2. These scene files should take a reasonable amount of time in current
> computers. This amount should be chosen so that it will not be in the order
> of a couple of seconds in a few years. For example they could take about 10
> minutes each in an 1.2GHz Athlon.

That's way too short. 
It should take at least two hours on the 1.4GHz Athlon with enough memory.
Provision should be made in the test protocol to change the scene again
when the reported time had reached less than 3 minutes on commonly
available.

And it should reflect the particularities of POV (so that doing
the same picture in other software(triangle based) should be
very difficult with the same short code and the same smoothness).


> 
>   3. The submission of entries should be controlled. Of course it's difficult
> to see if someone is just making a bogus entry if the numbers are credible,
> but there could be, for example, a "trusted entry system": Entries from
> trusted sources could get a mark showing that it's a trusted entry. Entries
> not having this mark may or may not be true (and this should be clearly
> stated in the page).

Easier, unless one has decided to really forge it: have pov output
a CRC as well as the statistics, and only accept capture from
the pov-output :-> If the CRC does not match, reject the entry :-<

Really, faked entries are of no concern for commonly available systems:
When a lot of P2/400 MHz gives a render time of x, the x/30 entry is
obviously faked.

Maybe the benchmark could summarize per kind of processor, with
the min,max and average, so reader may know what to expect from
a given kind of processor. (If the Celeron 333 metrics is 
1s/79s/119s , then there is obviously something strange, because a 
ratio of more than 100 for the same processor is difficult to explain
by the only change of the OS)

Personaly, I won't bother to have trusted/untrusted entries, nor
even to have any checking on value.

Post a reply to this message

That's a good idea, especially the idea of a whole benchmark suite, but
quite tricky on the other hand.  For example radiosity is still considered
as experimantal and differs between 3.1 and 3.5.  A benchmark would not be
complete without radiosity (since the intensive use of memory in a
radiosity scene is very important for performance) but you would not be
able to compare results from Povray 3.1 and 3.5.  Porting Povray 3.5 to
all platforms 3.1 currently runs on will take quite some time if it's
possible at all (since a C++ compiler is needed)

So what i would suggest is a general Povray 3.1 compatible benchmark scene
that's slower than skyvase and uses reasonable amount of memory.  

For an additional benchmark suite i would also use new features (and
radiosity) but only record results achieved with Povray 3.5 based compiles
since even if it's possible to render things with megapov results won't be
comparable.

Some conceptions for benchmark scenes i would suggest:

- reflection scene (low memory use, but slow because of highly complicated
reflections)
- complex csg scene (high memory use)
- radiosity scene (high memory use and very random access to memory)
- isosurface scene (testing performance in function evaluation)
- a large mesh scene (preferably macro generated, otherwise a large scene
file would be needed)
- a texture scene (basic geomentry but slow textures, would probably be
difficult to get this slow enough)

Christoph

-- 
Christoph Hormann <chr### [at] gmxde>
IsoWood include, radiosity tutorial, TransSkin and other 
things on: http://www.schunter.etc.tu-bs.de/~chris/

Post a reply to this message

Jérôme Grimbert <jer### [at] atosorigincom> wrote:
: Then, you would have multiple metrics. IMNSHO, it's a bad thing, because
: everyone will wants soon it's own test scene. 
: You should rather stick to a single scene, which should nevertheless
: be a complex-looking and lovely one.

  I'm not sure if I understand what do you mean.

  The idea with several scenes is that they measure different aspects of
the raytracing process. Of course the results of each scene should be
shown separately, although the total score could be the total sum of times.
  Of course the rendering time of all the scenes should be approximately
equal so that the results don't get biased towards one of them (ie. the one
which takes longer to render).
  There could be at least 4 different test files:

  1) A test for parsing speed: A pov-file which takes most of the time in
the parsing stage (doing something useful, not just idle loops) and a minimal
amount of time in the rendering part. This could be, for example, a scene
which places some thousands of spheres according to some very complicated
algorithm. The scene could take some memory at parsing time, but not at
render time.

  2) A test for heavy memory usage. The scene should take a considerable
amount of memory which is actively used while rendering. This could be
achieved with lots of light sources, huge amount of objects and so on.

  3) A test for raw raytracing speed: Parsing time and memory usage should
be negligible (ie. parsing time just a few seconds at max. and the scene
doesn't take any memory), but the raytracing part takes most of the time.

  4) A scene which combines all of the three.

  As I said, the total time of all the four scenes should be approximately
equal so that the results don't get biased.

  The idea behind this is that some computers are better in one of those than
in another, and this kind of test gives a good idea where is that computer
good at.
  One single scene can't test all of those things, and even if it does
(as in the 4th example), one can't see how well the computer performs in
the individual tasks.

: That's way too short. 
: It should take at least two hours on the 1.4GHz Athlon with enough memory.

  I think two hours is a bit overkill. Granted, 10 minutes may be too short,
but I think 2 hours is too much. I don't think people want to wait for hours
for this.
  Or perhaps if 4 pov-files are used, the total time could be about 2 hours,
which would mean a half-hour per file.

: Easier, unless one has decided to really forge it: have pov output
: a CRC as well as the statistics, and only accept capture from
: the pov-output :-> If the CRC does not match, reject the entry :-<

  That kind of CRC can be easily faked.

: Really, faked entries are of no concern for commonly available systems:
: When a lot of P2/400 MHz gives a render time of x, the x/30 entry is
: obviously faked.

  Of course, but I suppose that many people have the temptation to take away
a small but unnoticeable amount from the real rendering times (eg. if the
real rendering time was 35 minutes, they could just report 31 minutes and
no-one will notice). Unfortunately many people are like this; even if they
don't get anything from that (not even their name anywhere), they still tend
to "exaggerate" a bit to look better.

: Personaly, I won't bother to have trusted/untrusted entries, nor
: even to have any checking on value.

  Then there will just be a lot of bogus entries as in the current pov-bench,
which effectively destroys its usefulness.
  Even with the "average" thinking, absolutely no checking will still cause
problems. If the real average for a certain CPU would be 1 hour and someone
reports 20 times a rendering time of 1 second, that produces a substantial
change in the average.
  So there should definitely be some kind of checking. And I still think that
the "trusted source" method is good.

-- 
#macro N(D,I)#if(I<6)cylinder{M()#local D[I]=div(D[I],104);M().5,2pigment{
rgb M()}}N(D,(D[I]>99?I:I+1))#end#end#macro M()<mod(D[I],13)-6,mod(div(D[I
],13),8)-3,10>#end blob{N(array[6]{11117333955,
7382340,3358,3900569407,970,4254934330},0)}//                     - Warp -

Post a reply to this message

Christoph Hormann wrote:

> - radiosity scene (high memory use and very random access to memory)

High memory use would be bad. You can't know how much memory every
computer has installed and if someone runs out of memory, and has
to resort to disk swapping, it will negate their render time as
accurate. There are still people running machines with much less
than 128 megs and even those people with 128 megs installed have
the OS using up much of those resources.

I suggest the scene not use much more than 30 megs or so of memory.

-- 
Ken Tyler

Post a reply to this message

Warp wrote:
> 
> Jérôme Grimbert <jer### [at] atosorigincom> wrote:
> : Then, you would have multiple metrics. IMNSHO, it's a bad thing, because
> : everyone will wants soon it's own test scene.
> : You should rather stick to a single scene, which should nevertheless
> : be a complex-looking and lovely one.
> 
>   I'm not sure if I understand what do you mean.

There should be only one scene file. You are making it much too
complicated.

-- 
Ken Tyler

Post a reply to this message

Ken wrote:
> 
> High memory use would be bad. You can't know how much memory every
> computer has installed and if someone runs out of memory, and has
> to resort to disk swapping, it will negate their render time as
> accurate. There are still people running machines with much less
> than 128 megs and even those people with 128 megs installed have
> the OS using up much of those resources.
> 
> I suggest the scene not use much more than 30 megs or so of memory.

That's about what i thought, the main emphasis should lie on testing the
speed of memory access and having a scene of only 400k will not fulfil
this purpose (since most of the data will fit in cache on many systems).

Christoph

-- 
Christoph Hormann <chr### [at] gmxde>
IsoWood include, radiosity tutorial, TransSkin and other 
things on: http://www.schunter.etc.tu-bs.de/~chris/

Post a reply to this message

Ken wrote:
> 
> Warp wrote:
> >
> > Jérôme Grimbert <jer### [at] atosorigincom> wrote:
> > : Then, you would have multiple metrics. IMNSHO, it's a bad thing, because
> > : everyone will wants soon it's own test scene.
> > : You should rather stick to a single scene, which should nevertheless
> > : be a complex-looking and lovely one.
> >
> >   I'm not sure if I understand what do you mean.
> 
> There should be only one scene file. You are making it much too
> complicated.

Exactly, Ken :-)

The scene file should produce a beautiful image
          (so as to astonish the new users),
And it should be long enough to parse & render (I stick to my 2 hours on 
 a 1.4GHz Athlon (*))
  (at least, it should start to teach patience** to the new users,
 hence, it must be a beautiful image which looks complicated)

*: remember, it's 2 hours for a huge size of at least 1600x1200 with AA.
*, also: when a dual-athlon with 2GHz each will be available
  (dual athlon is already, 2GHz probably in a few months/years), hoping 
  a port of povray 3.5+ (4.0 ??) will be multiprocessor compatible 
 (using the full power, with thread and so), the LONG 2Hr would be reduced
 to only 40 minutes,
and on a server with quad-mobo would only take 20 minutes...
Try using a beowolf cluster of 64 of unexpensive 1GHz (by that time), 
and the theorical rendering drop to less than 3 minutes !
If you do the computation with a shorter scene, you will found yourself
with a precision problem sooner than you would have expected.

** I must confess, I started with DKBtrace 2.12 a long time ago 
and it took more than one week to render ntreal on a poor CPU (without FPU).
(no harddisk either, all was on floppy !). It was a time when one had
better to thing the scene rather than trying multiple preview.
Mosaic preview was a great thing then!

Post a reply to this message

On Mon, 20 Aug 2001 15:34:53 +0200, Jérôme Grimbert wrote:
>And it should be long enough to parse & render (I stick to my 2 hours on 
> a 1.4GHz Athlon (*))

But that would prevent people with real-world processors from participating
at all.  I hate to think how long it would take this P200 to run the 
benchmark, for example, but even a Duron 650 would have a tough time of it,
and that's a current processor.  I'm sure I'm not the only one who doesn't
expect to break the gigahertz barrier for at least another year or two.

-- 
#macro R(L P)sphere{L F}cylinder{L P F}#end#macro P(V)merge{R(z+a z)R(-z a-z)R(a
-z-z-z a+z)torus{1F clipped_by{plane{a 0}}}translate V}#end#macro Z(a F T)merge{
P(z+a)P(z-a)R(-z-z-x a)pigment{rgbf 1}hollow interior{media{emission 3-T}}}#end 
Z(-x-x.2x)camera{location z*-10rotate x*90normal{bumps.02scale.05}}

Post a reply to this message

Warp wrote:
> 
>   2. These scene files should take a reasonable amount of time in current
> computers. This amount should be chosen so that it will not be in the order
> of a couple of seconds in a few years. For example they could take about 10
> minutes each in an 1.2GHz Athlon.

Not enough, as others noted. I remember seeing entries like
several minutes for early Pentiums (that's the last time I
checked the bench pages) and several hours for 386.

I think nowadays it would be half an hour or more for around
1 GHz processors.

-- 
Adrien Beau - adr### [at] freefr - http://adrien.beau.free.fr
 Mes propos n'engagent que moi et en aucun cas mes employeurs

Post a reply to this message

Ken wrote:
> 
> There are still people running machines with much less
> than 128 megs and even those people with 128 megs installed have
> the OS using up much of those resources.

Then they have a problem. They shouldn't use an OS that eats
the vast majority of the installed RAM. Either they use an OS
that eats less memory, or intall more memory.

> I suggest the scene not use much more than 30 megs or so of memory.

That seems to be a good value. What's important is to be sure
it won't hold in the cache.

-- 
Adrien Beau - adr### [at] freefr - http://adrien.beau.free.fr
 Mes propos n'engagent que moi et en aucun cas mes employeurs

Post a reply to this message

Ron Parker wrote:
> 
> On Mon, 20 Aug 2001 15:34:53 +0200, Jérôme Grimbert wrote:
> >And it should be long enough to parse & render (I stick to my 2 hours on
> > a 1.4GHz Athlon (*))
> 
> But that would prevent people with real-world processors from participating
> at all. I hate to think how long it would take this P200 to run the
> benchmark, for example, but even a Duron 650 would have a tough time of it,
> and that's a current processor.  I'm sure I'm not the only one who doesn't
> expect to break the gigahertz barrier for at least another year or two.
> 

May I add that my current processor is a P200 underclocked for stability
at 180 with a lot of memory. If it took 2 Hours for a 1.4GHz, I would
expect my computer to take about two days. I do not see any problem about that.

If I were to just have a look at the image, whose size for the benchmark
should be bigger than 1600x1200, I could render it at 400x300, which would
be about 16 times faster (if only rendering was concerned), putting me
back in nearly realtime rendering (less than 3 hours).

We cannot have a reasonnable time for slow processor and enough 
provision for precision for fast futur processor and clustered things.

P.S. : I do not expect to upgrade before one full year, either.

Post a reply to this message

Adrien Beau wrote:
> 
> Ken wrote:
> >
> > There are still people running machines with much less
> > than 128 megs and even those people with 128 megs installed have
> > the OS using up much of those resources.
> 
> Then they have a problem. They shouldn't use an OS that eats
> the vast majority of the installed RAM. Either they use an OS
> that eats less memory, or intall more memory.

The vast majority of POV-Ray users are also Windows users. Windows
notoriously uses quite a bit of the system memory resources. Good
or bad this is a fact that must be considered.

-- 
Ken Tyler

Post a reply to this message

Ken wrote:
> 
> Adrien Beau wrote:
> >
> > Then they have a problem. They shouldn't use an OS that eats
> > the vast majority of the installed RAM. Either they use an OS
> > that eats less memory, or intall more memory.
> 
> The vast majority of POV-Ray users are also Windows users. Windows
> notoriously uses quite a bit of the system memory resources. Good
> or bad this is a fact that must be considered.

I know that of course. There was some underground rant in my
answer. :-)

SDRAM is cheap nowadays, so owners of quite-recent computers
(less than two years old?) should be able to upgrade without
bothering for the money. In Paris, 128 MB RAM is around the
same price as an audio CD. I think the "standard" amount of
RAM "should" be 256 MB.

And anyway, I think 30 MB memory-usage is a good value.

-- 
Adrien Beau - adr### [at] freefr - http://adrien.beau.free.fr
 Mes propos n'engagent que moi et en aucun cas mes employeurs

Post a reply to this message

"Warp" <war### [at] tagpovrayorg> wrote in message
news:3b810636@news.povray.org...
>   The idea behind this is that some computers are better in one of those
than
> in another, and this kind of test gives a good idea where is that computer
> good at.
>   One single scene can't test all of those things, and even if it does
> (as in the 4th example), one can't see how well the computer performs in
> the individual tasks.

Personally, I like that part.

>
> : That's way too short.
> : It should take at least two hours on the 1.4GHz Athlon with enough
memory.
>
>   I think two hours is a bit overkill. Granted, 10 minutes may be too
short,
> but I think 2 hours is too much. I don't think people want to wait for
hours
> for this.
>   Or perhaps if 4 pov-files are used, the total time could be about 2
hours,
> which would mean a half-hour per file.

Two hours for a benchmark seems a little excessive for me (especially on a
1.4gHz Athlon, which will make my 850mHz Duron seem like a slug...), but it
would make the benchmark last longer.  The problem with that, though, is
that in future years POV-Ray may well acquire new features / optimizations
which would mean we would need new test scenes.  We might have to rewrite
the test scenes before we even get to the 10 minute stage...

> : Really, faked entries are of no concern for commonly available systems:
> : When a lot of P2/400 MHz gives a render time of x, the x/30 entry is
> : obviously faked.
>
>   Of course, but I suppose that many people have the temptation to take
away
> a small but unnoticeable amount from the real rendering times (eg. if the
> real rendering time was 35 minutes, they could just report 31 minutes and
> no-one will notice). Unfortunately many people are like this; even if they
> don't get anything from that (not even their name anywhere), they still
tend
> to "exaggerate" a bit to look better.

I had never even thought of faking an entry.  It's a sad, sad world...
Seriously, if that's a problem, then write a small utility program that will
call POV with the appropriate options, grab the output and FTP / email the
results somewhere they can be accepted.  Unless the user wants to hack the
utility, no problem (I don't see why anyone would hack this, as it isn't
very interesting for people outside of this group, but then, I'm not most
people...)

...Chambers

Post a reply to this message

"Jérôme Grimbert" <jer### [at] atosorigincom> wrote in message
news:3B8111FD.BA95FF6D@atosorigin.com...
> *: remember, it's 2 hours for a huge size of at least 1600x1200 with AA.
> *, also: when a dual-athlon with 2GHz each will be available
>   (dual athlon is already, 2GHz probably in a few months/years), hoping
>   a port of povray 3.5+ (4.0 ??) will be multiprocessor compatible
>  (using the full power, with thread and so), the LONG 2Hr would be reduced
>  to only 40 minutes,
> and on a server with quad-mobo would only take 20 minutes...
> Try using a beowolf cluster of 64 of unexpensive 1GHz (by that time),
> and the theorical rendering drop to less than 3 minutes !
> If you do the computation with a shorter scene, you will found yourself
> with a precision problem sooner than you would have expected.

How many people do you know who have a Beowulf cluster at home?!?!?
I honestly know only one person who has a (home) computer in the gHz
range... and it's precisely 1 gHz.
Even though multiprocessor systems have been around for years, I still don't
know anyone who has one.

...Chambers

Post a reply to this message

Great conversation, all interesting thoughts, but here's another idea:
How about an iterative/recursive benchmark?  Write a small external utility
to call POV with custom scenes.  If POV returns too fast (say, under 5
minutes) then it creates a slightly more complex scene and tries it again.
It could even spit out different types of scenes (one where a number of
spheres are randomly distributed in a space, or a reflective-sphere floating
over a checkered plane with an area light that keeps getting more and more
lights in it, or one that gets more and more levels of textures, or inside a
reflective box and keep bumping up the max_recursion level).  For really
slow processors, it would stop reasonably soon and say "OK, that's the limit
of your processor", or, for really fast processors, it might have to keep
bumping things up.  The benchmark, then, would not be render time but scene
complexity.
Any thoughts?

...Chambers

Post a reply to this message

On Mon, 20 Aug 2001 16:08:17 +0200, Adrien Beau wrote:
>Ken wrote:
>> 
>> There are still people running machines with much less
>> than 128 megs and even those people with 128 megs installed have
>> the OS using up much of those resources.
>
>Then they have a problem. They shouldn't use an OS that eats
>the vast majority of the installed RAM. Either they use an OS
>that eats less memory, or intall more memory.

That's not allways an option, I find that people tend to have as 
much hardware as they can afford.  I have one machine here with 
Win 95 which has 32 M RAM, and if I get more memory it's going in 
the main box, not in the Win 95 machine. 

There will be people on this group who have only one machine and 
have only 32M RAM and no opportunity to do upgrades such as OS or 
memory. 

--
Cheers
Steve              email mailto:ste### [at] zeroppsuklinuxnet

%HAV-A-NICEDAY Error not enough coffee  0 pps. 

web http://www.zeropps.uklinux.net/

or  http://start.at/zero-pps

  6:35pm  up 37 days, 20:39,  2 users,  load average: 1.02, 1.03, 1.00

Post a reply to this message

Steve <ste### [at] zeroppsuklinuxnet> wrote:
: There will be people on this group who have only one machine and 
: have only 32M RAM and no opportunity to do upgrades such as OS or 
: memory. 

  When speaking about Windows, if you want more memory, you'll have to
_downgrade_ to an older version, not upgrade to a newer. :P

-- 
#macro N(D,I)#if(I<6)cylinder{M()#local D[I]=div(D[I],104);M().5,2pigment{
rgb M()}}N(D,(D[I]>99?I:I+1))#end#end#macro M()<mod(D[I],13)-6,mod(div(D[I
],13),8)-3,10>#end blob{N(array[6]{11117333955,
7382340,3358,3900569407,970,4254934330},0)}//                     - Warp -

Post a reply to this message

Here is my two bits:

I would prefer one scene file. That should be a scene that uses as many
of Pov's features as possible, perhaps with the exception of those
dependend of bitmaps, to avoid having to download large bitmap files
along with the benchmark scene file.

To get around the problem of a slow enough scene for state of the art
processors being too slow for older systems, I propose the following
system:

The benchmark should be expressed as the speed relative to a reference
system (like Andreas Tillner's ChessMark (see www.tabsnet.de)). On the
reference system (say a Athlon 1GHz, 64MB RAM, Windows, official PovRay
compile), the scene is rendered at several different resolutions (say
from 320x240 to 3200x2400). On fast system you choose a high resolution,
on slow systems a low resolution for running the benchmark. To avoid
timing inaccuracy, a minimum render time should be set. 

/Ib

Post a reply to this message

Heavily snipped from Ib Rasmussen's post:  (the following had stuff where there
are blank lines)

>I would prefer one scene file. That should be a scene that uses as many

>To get around the problem of a slow enough scene for state of the art

>compile), the scene is rendered at several different resolutions (say
>from 320x240 to 3200x2400). On fast system you choose a high resolution,
>on slow systems a low resolution for running the benchmark. To avoid

The problem with using a single file, and multiple resolutions to try to
compensate for different machines is that it only compensates for differences
in the rendering time.  If you've got a scene that takes several minutes for
the wonder-machine to parse, then my lowly P1-133 with 24Mb memory might take
?hours? (or some large  amount of time anyways) to parse it, even though I'm 
only doing the render at 320x240.

The same with memory issues.  128 Mb is becoming a fairly standard amount of
memory in a new machine.  So to really test how POV performs for large memory
accesses, you should use a reasonable chunk of that.  Which is going to drive
my machine into swapping, since memory is as much a function of number of
objects in the scene as it is of rendering resolution.

Ultimately, for accurate testing, what's probably needed is several catagories
of test files (parse time, memory usage, render time, etc) at several different
levels (from 486-66 to drool-worthy-super-machine).  

You could then run up the scale of increasingly-demanding tests until you got
a meaningfull time (say order 10 min), and post your results indicating both
what time, and which test.

Admittedly, this gets complicated, in several terms: creating various test
files, probably create a utility to automatically run them and post the results
(that or very good directions), finding a good way to display the results,
etc.

Jamie

Post a reply to this message

I think that in cases where the current benchmark is too slow you should
render it as an animation with 10 or 100 or even 1000 frames and then divide
the total render time with the number of frames rendered. All the frames
would be the same but that the whole point that makes this method work...
Memory wouldn't be very high, but who says the POV-Ray benchmark has to
measure memory speed anyway.

This solution would be by far the simplest and it would be fully comparable
to test that have already been performed.

...Just a different (and perhaps slightly boring) approach to the problem...

Rune
--
3D images and anims, include files, tutorials and more:
Rune's World:    http://rsj.mobilixnet.dk (updated June 26)
POV-Ray Users:   http://rsj.mobilixnet.dk/povrayusers/
POV-Ray Webring: http://webring.povray.co.uk

Post a reply to this message

> But that would prevent people with real-world processors from
participating
> at all.

I agree.

Jim

Post a reply to this message

Good idea!

Just my 2 cents concerning multiprocessing or clustered thingies: the
current problem (which has been the case for at least two years) is that
the fastest clusters are completely saturated by the amount of data to
move. I remember reading that being unable to even send blocks to part
of the nodes is quite common now.

The well known fact that the "usual" pvmpov has a 4s delay to wait for
stats when everything is over makes it clear that on such hardware
nobody will beat pvmegapov on skyvase, since the delay is 1s ;-)

I would suggest a reverse approach: ideally, I would like to get a 1%
accuracy on results from a *fast* cluster available when povray 3.5 is
out. This means an average timing of 100s on a 64 nodes cluster with
dual Athlon MP 1.6 GHz. Very roughly speaking, taking into account the
limited scalability of clusters, this means that Warp's 2 hours
suggestion would be perfect for this purpose. On the other hand this
means that the p200 box I'm currently using would take 2 days.
-- 
François DISPOT  --  http://www.wozzeck.net
      __  __ __  __  _
|  | /  \  /  / |_  /  |/
\/\/ \__/ /_ /_ |__ \_ |\

Post a reply to this message

There is a new (OK, not well known) POVRAY-Benchmark at
http://www.tabsnet.com/ waiting of your results. A overclocked Athlon
Thunderbird 1000@1620 MHz needs 00:05:21 to render the example chess2.pov
with the this parameter : povray -d -q9 +w800 +h600 +b1000 +r3 +a0.300 +v1
+ft . And there are some supercomputer at the list of Multi-CPU-Systems.
Every Benchmarkresult is very welcome.

Regards,
   Tilly,

Post a reply to this message

Good idea, and probably long overdue too.

I agree with Warp's original idea that there should be several different
types of scenes, including one all-rounder.  This would allow those who
favour a certain type of scene to make meaningful comparisons.  For
example, if someone likes scenes with lots of plants built from
enourmous meshes, then they may well be much more interested in seeing
the benchmarks for scenes with high memory usage, and high parsing times
(if e.g. their plants were procedurally created and placed).  And of
course the all-round scene is vital for an overall comparison.

Perhaps it is not so important to accomodate those with lower spec
computers.  I would assume that one of the main uses of a benchmarking
system is to compare the performance of higher-end systems, as the
results for these would be very helpful for anyone thinking of upgrading
(e.g. Athlon/P4 comparisons).  Of course, there is the case of those
deciding which OS to run on a lower spec machine that will be used for
rendering, but perhaps with enough results publicly available, they
could get their answer without having to do the test themselves.

As regards rendering time - it might be wise to keep it under 10 hours,
but not too far under so as to allow for future developments.  This
would be handy for leaving the computer on overnight whilst rendering
the scene.  Longer times would be difficult to trust, as people may well
have used the computer for other things during the render.  Slower
computers that had been playing MP3s at the same time would return
results that were not really worth anything.  Well, unless there was a
benchmark category for rendering whilst playing music :-)

Oh, I guess we should decide pretty soon, as someone's going to have to
produce the scenes in the first place, and it would make sense for them
to be in the 3.5 distribution.

Post a reply to this message

Warp wrote:

>
>
>   The idea behind this is that some computers are better in one of those than
> in another, and this kind of test gives a good idea where is that computer
> good at.
>   One single scene can't test all of those things, and even if it does
> (as in the 4th example), one can't see how well the computer performs in
> the individual tasks.
>

I agree with this*, an aggregate result could easily hide some interesting
comparisons, which brings me to my footnote:
* what is the actual purpose(s) of a benchmark scene?

Jim

Post a reply to this message

Francois Dispot <woz### [at] club-internetfr> wrote:
: Warp's 2 hours suggestion

  It wasn't my suggestion.

  I think 2 hours is too much.

-- 
#macro N(D,I)#if(I<6)cylinder{M()#local D[I]=div(D[I],104);M().5,2pigment{
rgb M()}}N(D,(D[I]>99?I:I+1))#end#end#macro M()<mod(D[I],13)-6,mod(div(D[I
],13),8)-3,10>#end blob{N(array[6]{11117333955,
7382340,3358,3900569407,970,4254934330},0)}//                     - Warp -

Post a reply to this message

On 20 Aug 2001 14:48:33 -0400, Warp wrote:
>Steve <ste### [at] zeroppsuklinuxnet> wrote:
>: There will be people on this group who have only one machine and 
>: have only 32M RAM and no opportunity to do upgrades such as OS or 
>: memory. 
>
>  When speaking about Windows, if you want more memory, you'll have to
>_downgrade_ to an older version, not upgrade to a newer. :P

By upgrade I didn't mean to a later version of Windows, nuff said. 

--
Cheers
Steve              email mailto:ste### [at] zeroppsuklinuxnet

%HAV-A-NICEDAY Error not enough coffee  0 pps. 

web http://www.zeropps.uklinux.net/

or  http://start.at/zero-pps

 12:15am  up 38 days,  2:19,  3 users,  load average: 1.04, 1.10, 1.14

Post a reply to this message

Steve <ste### [at] zeroppsuklinuxnet> wrote:
: nuff said. 

  By the way, what does this mean? I have seen it many times but I have
no idea...

-- 
#macro N(D,I)#if(I<6)cylinder{M()#local D[I]=div(D[I],104);M().5,2pigment{
rgb M()}}N(D,(D[I]>99?I:I+1))#end#end#macro M()<mod(D[I],13)-6,mod(div(D[I
],13),8)-3,10>#end blob{N(array[6]{11117333955,
7382340,3358,3900569407,970,4254934330},0)}//                     - Warp -

Post a reply to this message

Greetings everyone (yes, I have been lurking out here for a bit... good
to be back after a long absense); Topic of conversation finally drew me
out of my shell for a brief de-lurk.

Ben has an idea here that I'd like to expand only with a twist:
namely, iteration.

Thing is, instead of parsing and rendering a scene (or several scenes) for
a benchmark, I think a couple of the problems mentioned in this thread might
be solved by having built-in timed iteration loops. Yes, I know, none of us
wants to see a bloat of POV itself in the official release, but perhaps an
unofficial version with the iteration patches built in could be distributed
for those who wanted to benchmark POV on their own system. Picture it
like this:

First, patch looks for a standard .pov file "benchmark.pov" (If it doesn't
exist,the testing patch could autogenerate it to ensure you're using the
right scene). Next, the processor repeatedly parses that file (which would be
chosen to contain things that really give the parsing routines a workout)
for a fixed period of time (relatively short, but enough to get a
statisticly valid reading), starting over each time the file is completed
(if it is!) so that even mega powerful machines will have something to do
for the full parse test. Benchmark the parsing capabilities of the machine
based on how many lines were parsed in the testing period.

Then choose a standard set of POV-Ray's internal functions to test, say,
>intersection testing of...
   a sphere,
   a julia fractal,
   a modestly complex isosurface,
   sequentially arranged transparent and/or reflective objects
>sampling of media,
>calculation of a few complex texture patterns
>(and whatever other features you felt it a good idea to include in a
benchmark.

Each of these functions would be called not in actual scene, but *as if*
you were calling them for a scene, in a timed loop, with the number of
calls completed within the predetermined timeframe being recorded as a
benchmark for that function. This way instead of using a factor such as
pixels per second, or the total amount of time for a given test scene, you
would measure how fast the test platform processes various commonly accessed
functions which contribute to the rendering of a scene. Among other things,
this would allow patch writers to plug their newer, presumably more
streamlined versions of existing functions into the iterative testing code
and see if in fact their revised source code (or choice of compiler) truly
made a difference to each given aspect of raytracing.

Presumably any characteristic of POV you wanted to benchmark could have
an iterative quantifier test designed and included in the patch, and since
the time factor for which the test is run would not vary regardless of the
system you run it on, you'd be measuring actual performance directly rather
than trying to infer performance from how long a particular scene took to
parse/trace.


Charles
----
The Silver Tome ::  http://www.silvertome.com
0x06: I am NOT a RESOURCE. I am a free man!!
0x02: Aaaahahahahahahaha...
--
"Ben Chambers" <bdc### [at] hotmailcom> wrote in message
news:3b813537@news.povray.org...
> Great conversation, all interesting thoughts, but here's another idea:
> How about an iterative/recursive benchmark?  Write a small external utility
> to call POV with custom scenes.  If POV returns too fast (say, under 5
> minutes) then it creates a slightly more complex scene and tries it again.
> It could even spit out different types of scenes (one where a number of
> spheres are randomly distributed in a space, or a reflective-sphere floating
> over a checkered plane with an area light that keeps getting more and more
> lights in it, or one that gets more and more levels of textures, or inside a
> reflective box and keep bumping up the max_recursion level).  For really
> slow processors, it would stop reasonably soon and say "OK, that's the limit
> of your processor", or, for really fast processors, it might have to keep
> bumping things up.  The benchmark, then, would not be render time but scene
> complexity.
> Any thoughts?
>
> ...Chambers
>
>

Post a reply to this message

"Christoph Hormann" <chr### [at] gmxde> wrote in message
news:3B80FDBA.D4B09876@gmx.de...
Porting Povray 3.5 to
> all platforms 3.1 currently runs on will take quite some time if it's
> possible at all (since a C++ compiler is needed)
>

    Is this true ? The fact that no-ones picked it up implies it is, I
guess, but I thought POV wasn't going C++ until 4.0...

--
Scott Hill.
Software Engineer.
E-Mail        : sco### [at] innocentcom
Pandora's Box : http://www.pandora-software.com

*Everything in this message/post is purely IMHO and no-one-else's*

Post a reply to this message

Warp wrote:
> 
> Steve <ste### [at] zeroppsuklinuxnet> wrote:
> : nuff said.
> 
>   By the way, what does this mean? I have seen it many times but I have
> no idea...

nuff said <-> enough said (on this subject).

-- 
Ken Tyler

Post a reply to this message

Warp wrote:
> 
> Steve <ste### [at] zeroppsuklinuxnet> wrote:
> : nuff said.
> 
>   By the way, what does this mean? I have seen it many times but I have
> no idea...
> 

I always thought of it as meaning "enough said" but I could be wrong.

Brendan

Post a reply to this message

Warp wrote in message <3b810636@news.povray.org>...
>Jérôme Grimbert <jer### [at] atosorigincom> wrote:
>: That's way too short.
>: It should take at least two hours on the 1.4GHz Athlon with enough
memory.
>
>  I think two hours is a bit overkill. Granted, 10 minutes may be too
short,
>but I think 2 hours is too much. I don't think people want to wait for
hours
>for this.
>  Or perhaps if 4 pov-files are used, the total time could be about 2
hours,
>which would mean a half-hour per file.


I'll repeat my earlier suggestion for a benchmark that will work equally
well on all systems: A benchmark image that spends almost all the time
actually rendering.  Fast systems should render at high resolutions, while
slow systems should render it at low resolutions.  The render time should
then be normalized to calculate how long that machine would have taken to
render the scene at some incredibly huge resolution (say, 40,000 x 30,000).

Examples:
A PII-400 renders at 640x480 in 652 seconds.  Since the area rendered is
1/3096.25 of the benchmark area, the time is multiplied by 3096.25 to get a
benchmark of 2,546,875, the estimated time to render the scene at
40000x30000.

A PIII-800 renders at 1024x768 in 750 seconds.  Since the area rendered is
1/1525.88 of the benchmark area, the time is multiplied by 1525.88 to get a
benchmark of 1,144,409.

A Beowulf cluster renders at 4000x3000 in 817 seconds.  Since the area is
1/100 of the benchmark area, the time is multiplied by 100 to get a
benchmark of 81,700.

--
Mark

Post a reply to this message

"Charles" <cfu### [at] enternet> wrote in message
news:3b819ecd@news.povray.org...
> <snip>

Well, it's interesting, but one of the things about rendering a full scene
is that you see the operations in action, ie, pseudo-random order thrashes
any looping speedups.  With modern processors, though (esp. those with big
caches) I don't know if this is really an issue.  Anyone else know?  Also, I
think it would be better just to write a separate utility which calls POV,
for space's sake.

Of course, there're probably as many ideas about how to do this as there are
people reading this thread... if not more... :)

...Chambers

Post a reply to this message

Scott Hill wrote:
> 
>     Is this true ? The fact that no-ones picked it up implies it is, I
> guess, but I thought POV wasn't going C++ until 4.0...
> 

I think it will be mainly plain C, but some parts require a C++ compiler.
IIRC Thorsten mentioned this quite some time ago.

Christoph

-- 
Christoph Hormann <chr### [at] gmxde>
IsoWood include, radiosity tutorial, TransSkin and other 
things on: http://www.schunter.etc.tu-bs.de/~chris/

Post a reply to this message

Mark Wagner wrote:
>
> actually rendering.  Fast systems should render at high resolutions, while
> slow systems should render it at low resolutions. 

What about features that use some treshold to decide whether to
make more computations or not?  Anti aliasing, reflections, media,
focal blur, radiosity etc. The pixels/second reading wouldn't be
same at different resolutions.


_____________
Kari Kivisalo

Post a reply to this message

> > : nuff said.
> > 
> >   By the way, what does this mean? I have seen it many times but I have
> > no idea...
> 
> nuff said <-> enough said (on this subject).

I've also seen it used to imply 'I've said all I'm going to say on this 
subject'

Bye for now,
     Jamie.

Post a reply to this message

"Jamie Davison" <jam### [at] ntlworldcom> wrote in message
news:MPG.15ec4a7dcdd94ec79899c1@news.povray.org...
> > > : nuff said.
> > >
> > >   By the way, what does this mean? I have seen it many times but I
have
> > > no idea...
> >
> > nuff said <-> enough said (on this subject).
>
> I've also seen it used to imply 'I've said all I'm going to say on this
> subject'

Darn, because I saw a chance to ask why gh=f anyhow.  <g>
Can't think of anyone else around here with a "gh" in their name, probably
is though ("though"... no "f" sound there).  Ha ha.  Pay no attention to me.
I've been asked if Hughes is pronounced Hufes before, believe it or not.

Bob H.

Post a reply to this message

"Warp" <war### [at] tagpovrayorg> wrote in message news:3b819dc2@news.povray.org...
> Steve <ste### [at] zeroppsuklinuxnet> wrote:
> : nuff said.
>
>   By the way, what does this mean? I have seen it many times but I have
> no idea...

Enough said - it comes from Marvel Comics, Stan Lee's Soapbox back in the '60s
IIRC

(f'ups set to o.t.)

Post a reply to this message

On Tue, 21 Aug 2001 01:41:08 +0100, "Scott Hill"
<nos### [at] nospamthanks> wrote:

>    Is this true ? The fact that no-ones picked it up implies it is, I
>guess, but I thought POV wasn't going C++ until 4.0...

Some parts of 3.5 are in C++, and it does require a C++ compiler to
build.


Peter Popov ICQ : 15002700
Personal e-mail : pet### [at] vipbg
TAG      e-mail : pet### [at] tagpovrayorg

Post a reply to this message

On 2001-08-20 12:56, Ken <tyl### [at] pacbellnet> wrote:
> Christoph Hormann wrote:
> 
>> - radiosity scene (high memory use and very random access to memory)
> 
> High memory use would be bad. You can't know how much memory every
> computer has installed and if someone runs out of memory, and has
> to resort to disk swapping, it will negate their render time as
> accurate.

Their render times are still accurate. Render time does depend on the
amount of memory you have (and the quality of the swapping algorithm and
the speed of the disks) for large scenes. Memory is as much (or even
more) constricting to my scenes than CPU power is. Therefore a benchmark
suite should contain scenes with varying amounts of memory.

> There are still people running machines with much less
> than 128 megs and even those people with 128 megs installed have
> the OS using up much of those resources.

Right. This means that they cannot expect good performance when
rendering large scenes. Just like someone with a slow CPU cannot expect
good performance when rendering scenes which need a lot of CPU power.

	hp

-- 
   _  | Peter J. Holzer    | Oder glaubst du "Bugtraq" waere eine
|_|_) | Sysadmin WSR       | Science-Fiction TV-Serie ueber Schaben
| |   | hjp### [at] hjpat         | im Weltall?
__/   | http://www.hjp.at/ |	-- Juergen P. Meier in dcsm

Post a reply to this message

On 2001-08-20 10:59, Jérôme Grimbert <jer### [at] atosorigincom> wrote:
> Warp wrote:
>> 
>>   1. It has more than one scene file, each one benchmarking its own important
>> area in rendering, for example one for raw raytracing speed, one
>> which takes long to parse, one which uses lots of memory, etc.
>
> Then, you would have multiple metrics. IMNSHO, it's a bad thing,

No, it's a good thing, because benchmarks with only a single metric are
almost useless in comparing systems.

> because everyone will wants soon it's own test scene.

The number of test scenes should be small enough that a reasonable
number of systems will be tested with each. But there should be enough
to highlight different aspects of systems, and having several scenes
also makes it easier to replace them over time without invalidating all
results.

	hp

-- 
   _  | Peter J. Holzer    | Oder glaubst du "Bugtraq" waere eine
|_|_) | Sysadmin WSR       | Science-Fiction TV-Serie ueber Schaben
| |   | hjp### [at] hjpat         | im Weltall?
__/   | http://www.hjp.at/ |	-- Juergen P. Meier in dcsm

Post a reply to this message

On 2001-08-20 15:59, Ben Chambers <bdc### [at] hotmailcom> wrote:
> Even though multiprocessor systems have been around for years, I still don't
> know anyone who has one.

I had one (a dual 90 MHz Pentium). We also have several at the
institute (they are a bit newer, though :-).

	hp


-- 
   _  | Peter J. Holzer    | Oder glaubst du "Bugtraq" waere eine
|_|_) | Sysadmin WSR       | Science-Fiction TV-Serie ueber Schaben
| |   | hjp### [at] hjpat         | im Weltall?
__/   | http://www.hjp.at/ |	-- Juergen P. Meier in dcsm

Post a reply to this message

In article <slr### [at] fwicom> , ron### [at] povrayorg (Ron
Parker) wrote:

> But that would prevent people with real-world processors from participating
> at all.  I hate to think how long it would take this P200 to run the
> benchmark, for example, but even a Duron 650 would have a tough time of it,
> and that's a current processor.  I'm sure I'm not the only one who doesn't
> expect to break the gigahertz barrier for at least another year or two.

But skyvase.pov took hours to render too when it was first created ...

IMO, the longer it takes the better, up to a certain extend of course.  I
would say 48 or even more hours on a slow (i.e. P90 system) would still do
the job for most people.  One could for example let it render on a sunny
weekend without suffering from computer withdrawal while POV-Ray blocks the
system ;-)

    Thorsten

____________________________________________________
Thorsten Froehlich, Duisburg, Germany
e-mail: tho### [at] trfde

Visit POV-Ray on the web: http://mac.povray.org

Post a reply to this message

Thorsten Froehlich wrote:

> But skyvase.pov took hours to render too when it was first created ...
> 
> IMO, the longer it takes the better, up to a certain extend of course.  I
> would say 48 or even more hours on a slow (i.e. P90 system) would still do
> the job for most people.  One could for example let it render on a sunny
> weekend without suffering from computer withdrawal while POV-Ray blocks the
> system ;-)

I coudn't stand not being connected to the internet for that long !
As soon as I did it would skew the results.

-- 
Ken Tyler

Post a reply to this message