On Thu, 19 Aug 1999 01:55:03 -0700, Ken <tyl### [at] pacbellnet> wrote:

>...but I'm sure there must be some routines
>in Pov that could be used for a reliable benchmark. Probably a combination
>of math functions, high order primitives, and complex surface finishes that
>require large numbers of ray intersection tests. All of this can still be
>done without the need to consume large quantities of memory so that disk
>swapping can be avoided on even under equipped machines.

  Sounds like a fun project for the rest of the summer. It might be best to
omit experimental features such as radiosity, as their implementation may
change too soon for the benchmark to gain a following. The scene should be
complex enough to give the faster machines a workout, yet not so 'tough' as
to render (pun intended) the slower machines useless for several days. I
think it should also be aesthetically pleasing and not just a loop-generated
scene with thousands of reflective and refractive objects.

-- 
Alan
--------------------------------------------------------------------
http://www.povray.org - Home of the Persistence of Vision Ray Tracer
news.povray.org - where POV-Ray enthusiasts around the world can get
together to exchange ideas, information, and experiences with others
--------------------------------------------------------------------

Post a reply to this message

Hi Ken:

In essence, you are asking if I believe meaningful comparison is possible.
I sure do.

I also believe that it's possible to properly compare apples and oranges--
but only by asking the RIGHT question!   'Which is better for your health?'
is not a meaningful question.   'Which is better if one has scurvey?' is
measurable and thus answerable.

Similarly, 'Which is the more powerful CPU?' should be sharpened by: Power
to do WHAT?  CPU A and B may be virtually equal as word processors, while A
could blow the socks off B while searching for prime numbers.   A question
about "raw computing power" is usually a raw, unripened question.

Similarly, "true test of performance" is an unfinished premise.   It
requires the "performing what" to be complete.    Even that requires
judgment.   Ten percent better performance at "calculating spreadsheets" is
a silly reason to buy a machine if it only 'calculates' a few minutes a
day-- better to buy based upon color and cute buttons.  Yet, that same
superiority in a worker-bee machine can be worth baskets o' bucks.

I would **never** exclude the comparison of different classes of machines--
those are *exactly* the kinds of comparisons we MUST make to choose hardware
for a particular job.

My invocation of benchmark caveats was more cautionary than exclusionary.
Use them with full awareness of what we really looking at in the numbers.
Assume nothing.

I had a blast going through the benchmark sites before deciding upon
hardware.  Looked at everything from 66MHz on up.  In the end,  I chose a
path which entails CPUs and motherboard which are not quite the most
efficient or the best immediate value per MFLOP-- but they have a 100%+
performance upgrade path and attractive motherboard features.

Matt

>
>Matt Swarm wrote:
>
>> When we shift the focus from the work a particular machine will do to
>> TESTING CPUs and machines in general, Ken, it seems to me than NM's
>> criticism becomes more applicable.   For example, if an x86 compiler used
on
>> POV source is inferior to an Alpha compiler, the Alpha might look 50%
faster
>> for certain POV programs.  Clearly, we are not just testing hardware.
>>
>> Matt
>
>  I won't argue your point... BUT... well ok. Then the only true test
>of your performance would be when it is compared to a similar system
>architecture and platform running at a specific speed. Any other
>comparison would simply imply that your machine is different than
>someone else's and your results would therefore be misleading. It
>that what you are implying ?
>
>--
>Ken Tyler
>
>See my 700+ Povray and 3D Rendering and Raytracing Links at:
>http://home.pacbell.net/tylereng/index.html

Post a reply to this message

>The scene should be
>complex enough to give the faster machines a workout, yet not so 'tough' as
>to render (pun intended) the slower machines useless for several days. I
>think it should also be aesthetically pleasing and not just a
loop-generated
>scene with thousands of reflective and refractive objects.
>

Several have commented on this issue.

I agree on the need, but fear that a conventional program approach is
destined to fail.

The problem is that the gap between the slowest machines and the fastest is
growing nonlinearly.   What makes this so is the 'Grove effect' plus growth
in distributive and cluster computing.

I propose a Smart Benchmark, which senses how long a routine is taking, and
terminates after a set period, projecting the time for completion of the
'full' program-- whatever the machine.   All this code is (or SHOULD be)
deterministic, nonrandom, so extrapolation should be straightforward.

My benchmarking experience is limited-- there may be things lke this out
there, even if non-POV.

Matt


>--
>Alan
>--------------------------------------------------------------------
>http://www.povray.org - Home of the Persistence of Vision Ray Tracer
>news.povray.org - where POV-Ray enthusiasts around the world can get
>together to exchange ideas, information, and experiences with others
>--------------------------------------------------------------------
>

Post a reply to this message

>The scene should be
>complex enough to give the faster machines a workout, yet not so 'tough' as
>to render (pun intended) the slower machines useless for several days. I
>think it should also be aesthetically pleasing and not just a
loop-generated
>scene with thousands of reflective and refractive objects.
>

Several have commented on this issue.

I agree on the need, but fear that a conventional program approach is
destined to fail.

The problem is that the gap between the slowest machines and the fastest is
growing nonlinearly.   What makes this so is the 'Grove effect' plus growth
in distributive and cluster computing.

I propose a Smart Benchmark, which senses how long a routine is taking, and
terminates after a set period, projecting the time for completion of the
'full' program-- whatever the machine.   All this code is (or SHOULD be)
deterministic, nonrandom, so extrapolation should be straightforward.
Maybe an animation, with option to deduct for disk access times.

My benchmarking experience is limited-- there may be things lke this out
there, even if non-POV.

Matt

Post a reply to this message

Matt Swarm wrote in message <37bc7173@news.povray.org>...
>>The scene should be
>>complex enough to give the faster machines a workout, yet not so 'tough'
as
>>to render (pun intended) the slower machines useless for several days. I
>>think it should also be aesthetically pleasing and not just a
>loop-generated
>>scene with thousands of reflective and refractive objects.
>>
>
>Several have commented on this issue.
>
>I agree on the need, but fear that a conventional program approach is
>destined to fail.
>
>The problem is that the gap between the slowest machines and the fastest is
>growing nonlinearly.   What makes this so is the 'Grove effect' plus growth
>in distributive and cluster computing.
>
>I propose a Smart Benchmark, which senses how long a routine is taking, and
>terminates after a set period, projecting the time for completion of the
>'full' program-- whatever the machine.   All this code is (or SHOULD be)
>deterministic, nonrandom, so extrapolation should be straightforward.
>Maybe an animation, with option to deduct for disk access times.


How about an image where the render time scales linearly with image area?
This way, you simply render the image at what you consider an appropriate
resolution for your machine (320x240 for a 486/33, 320000x240000 for a
supercomputer, etc) and then apply a correction factor for the area (the
486/33 in this example would have a correction factor of x1000000)?

Mark

Post a reply to this message

>>I propose a Smart Benchmark, which senses how long a routine is taking,
and
>>terminates after a set period, projecting the time for completion of the
>>'full' program-- whatever the machine.   All this code is (or SHOULD be)
>>deterministic, nonrandom, so extrapolation should be straightforward.
>>Maybe an animation, with option to deduct for disk access times.
>
>
>How about an image where the render time scales linearly with image area?
>This way, you simply render the image at what you consider an appropriate
>resolution for your machine (320x240 for a 486/33, 320000x240000 for a
>supercomputer, etc) and then apply a correction factor for the area (the
>486/33 in this example would have a correction factor of x1000000)?
>
>Mark


Interesting notion- certainly simpler than a tricky program.  How might this
work?

We have two machines, say, Trusty Rusty (486)  and the Swarm Machine I (20
500 MHz Celerons).

Q:   The clock speeds on Swarm add up to about 300 times Rusty.  Which image
size would we select, perhaps, for the Swarm Machine?

Q:   What correction factor would we then apply?

Q:   While it's true that image 1000X by 1000Y is 1000000 times greater in
area than image XY, is it also true that a THEORETICAL idealized machine
rendering in POV will also be 1000000 times longer?   It's true that area
increases by the square-- what worries me here is that (so far as I know) we
are doing some computations in THREE dimensions.  Might not some portions of
the image factor up by the CUBE of the increase instead of merely the
square?  Or worse?

Looking forward to your thoughts.

Matt

Post a reply to this message

Matt Swarm wrote in message <37bd15a0@news.povray.org>...
>
>
>>>I propose a Smart Benchmark, which senses how long a routine is taking,
>and
>>>terminates after a set period, projecting the time for completion of the
>>>'full' program-- whatever the machine.   All this code is (or SHOULD be)
>>>deterministic, nonrandom, so extrapolation should be straightforward.
>>>Maybe an animation, with option to deduct for disk access times.
>>
>>
>>How about an image where the render time scales linearly with image area?
>>This way, you simply render the image at what you consider an appropriate
>>resolution for your machine (320x240 for a 486/33, 320000x240000 for a
>>supercomputer, etc) and then apply a correction factor for the area (the
>>486/33 in this example would have a correction factor of x1000000)?
>>
>>Mark
>
>
>Interesting notion- certainly simpler than a tricky program.  How might
this
>work?
>
>We have two machines, say, Trusty Rusty (486)  and the Swarm Machine I (20
>500 MHz Celerons).
>
>Q:   The clock speeds on Swarm add up to about 300 times Rusty.  Which
image
>size would we select, perhaps, for the Swarm Machine?


This is a matter of personal preference, depending on many factors,
including how fast the computer is and how much time you can afford to have
the computer spend rendering the image.  However, I would recommend an image
size that would take at least five minutes to render, and preferably longer.

>Q:   What correction factor would we then apply?


Multiply the resulting time by (baseline image area/rendered image area) to
get the time corrected for the image area.

>Q:   While it's true that image 1000X by 1000Y is 1000000 times greater in
>area than image XY, is it also true that a THEORETICAL idealized machine
>rendering in POV will also be 1000000 times longer?   It's true that area
>increases by the square-- what worries me here is that (so far as I know)
we
>are doing some computations in THREE dimensions.  Might not some portions
of
>the image factor up by the CUBE of the increase instead of merely the
>square?  Or worse?


If antialiasing is not used, for sufficiently large images the only change
is in the number of rays traced, and since for large images, adjacent pixels
fire rays that follow similar pathes, and thus take similar amounts of time
to trace.  This is only true if the parsing time is negligable for all
machines involved, or is not counted for the total time, as parsing time is
independant of image resolution.

Mark

Post a reply to this message