And to add to this...

Recently your website used POV-Ray 3.6 for comparison of recent AMD and
Intel processors.  You end you discussion of POV-Ray as a benchmark by
implying that our code has been "hand-tweaked to favor one CPU".  The source
code of POV-Ray 3.6 is available on our website for download.  As such, it
would have been easy to determine that such claim is simply false.  Neither
the Windows nor the Linux version of POV-Ray contain any code specific to
any CPU at all.

You further imply that the benchmark scene would be unsuitable as it "fails
to demonstrate real world performance".  First of all, note that the
benchmark scene did not change from POV-Ray version 3.5 released over two
years ago to version 3.6 released a few weeks ago.

You also assume that demo scenes included with POV-Ray would demonstrate
such performance.  As stated in the user manual, the demo scenes included
with POV-Ray are there for educational purposes only.  As such, they
demonstrate only one or so very specific features of POV-Ray, consequently
using only a tiny portion of the POV-Ray codebase.

Unfortunately you decided to using custom and undisclosed setting for
rendering the benchmark scene as well as the other test scenes.  You also
decided to not specify with eight scenes of which fifty scenes you selected.
As such, nobody can easily reproduce your results.  However, we did render
the scenes from the "advanced" directory as they are those taking longest to
render.

The scene from the "advanced" directory we found taking longest to
render was "mediasky.pov".  This scene, as its name implies also shows only
one single feature, media.  As such, it heavily depends on the performance
of the noise function code, which is very small and tends to favor systems
with a fast L1 cache because it is very tiny.  The noise function is
certainly important to create a realistic scene, but it is never found
exclusively in any "real world" scene.

The scene taking second-longest to render is stackerday.pov and its "night"
version stackernight.pov takes about half the time to render.  There are
some other "stacker*.pov" variations all sharing  the same characteristics
and varying only in the coloring.  All these scenes contain may boxes and
text, which are only two of the 24 geometric primitives supported by
POV-Ray. The algorithm to intersect a ray with a box requires many
unpredictable comparison operation and branches, and as such processors with
a very long pipeline will perform poorly when running this algorithm.  As
such, the processor will be waiting to find out what to do next most of the
time while the pipeline stalls.  Even worse, they make heavy use of our
experimental radiosity feature.  As such, the stacker-scenes are very poor
choice to use as benchmark.

We determined "balcony.pov" to take fourth-longest to render.  This scene is
much closer to a suitable benchmark scene than all other scenes discussed
until now.  It has a fair selection of geometric primitives and uses several
other features.  As such, it does exercise many different parts of the
POV-Ray codebase.  Yet, its major drawback is that it uses "radiosity", a
feature we explicitly marked as experimental (a warning is shown
automatically if radiosity is used in a scene).

Rounding off the top-five is "optics.pov".  This scene demonstrates a single
POV-Ray feature, photon mapping.  The photon mapping algorithm requires a
lot of searching and sorting and in general tends to be optimized well by
current compilers and raw clock speed matters the most for this scene.  Most
of this simple scene should fit into the L2 cache because it only uses a
small amount of photons sufficient for such a specific demo.  Obviously this
is not a suitable benchmark scene.

Sixth longest to render was "landscape.pov".  It demonstrates a single
feature, the height-field object.  It does not even contain advanced
lighting or anything else fancy.  Only a tiny portion of the POV-Ray
codebase will be needed to render this scene.

Seventh longest took "glasschess.pov".  This scene uses ten different
geometric primitives, but it does not use noteworthy texturing.  Its goal is
to demonstrate refraction and reflections.  It also tends to fit entirely
into the L2 cache.  As such, it is not suitable to benchmark a system as a
whole, and its compactness certainly is not representative of the average
POV-Ray scene today.

Eighth was "stackertransp.pov", already discussed as part of the other
"stacker*.pov" scenes.

Ninth was "isocacti.pov", which demonstrates the isourface geometric
primitive together with a huge amount of cones.  Yet, the cones hardly
contribute anything the the overall computation, but they do push the scene
size well beyond the L2 cache size. Nevertheless, isosurfaces use a virtual
machine to evaluate user-supplied functions and as such this has little to
do with ray-tracing.  As such, the scene can say almost nothing about other
POV-Ray ray-tracing performance and consequently is an unsuitable benchmark.

The tenth longest-tracing scene was "newdiffract.pov".  This scene again
uses only almost only photons.  However, unlike "optics.pov", it uses many
more photons.  Thus, random memory access dominates in this scene when
searching for photons.  Still, the photons are only relevant in small
portions of the scene, and as such while there are many more photons, they
contribute much less to the scene.  Thus, apart from a bit of photon use,
this scene only contains few primitives and renders very quickly because it
is fairly simple.  It does certainly not represent any general scene, but is
only a demo to show diffraction.  As such, it exercises only a small part of
the POV-Ray codebase and is unsuitable as a benchmark scene.

Now, it should be noted that the first four longest-rendering scenes from
the "advanced" directory take already about as long as your eight scenes
relative to "benchmark.pov" when using the same settings.  Thus, unless you
specify the scenes you used to generate your results, they lack any
credibility.

Of course, there are many more demo scenes included with POV-Ray, but
without exception, all other demo scenes show only exactly a single POV-Ray
feature to help learning to use POV-Ray.  Thus, if you used any of those
scene for your benchmarking as part of the eight or fifty scenes you
mention, you can be certain you did not generate representative numbers at
all as you at most benchmarked performance of eight - which equals just one
third - different geometric primitives supported by POV-Ray.

And even if you ran eight of the top ten longest-rendering "advanced"
scenes, you encountered all the problems outlined above.  Consequently, your
benchmarking was simply flawed.

Of course, we do not claim "benchmark.pov" is the perfect benchmark, but it
does provide a balance between the many features used to create a great real
world POV-Ray scene.  Any claim we created a benchmark scene that is
intentionally or due to incompetence biased towards Intel processors has no
merit.  To the contrary, even two weeks prior to your article we had already
made available a beta version of POV-Ray 3.6 for Windows XP 64-bit Edition,
even adding support AMD 64-bit processor extensions!  Nevertheless, with the
32-bit version of POV-Ray runs as good as possible on AMD processors.

Consequently, we would like to suggest you refrain from making baseless
accusations that the POV-Team favors Intel processors or is incompetent at
creating a suitable benchmark.  Instead, we challenge you to provide the
actual scenes and settings used to generate the "benchmark" results you
claim more accurately reflect the performance apparently not even AMD and
Intel expect.

Sincerely,

    Thorsten Froehlich, POV-Team



PS: The term "codebase" refers to the compiled program code of POV-Ray and
has absolutely nothing to do with a "database" for which you confused it in
your article.  We are currently working to reworded out website to no long
use the technical term "codebase" as it does seem to confuse non-computer
scientists.

Post a reply to this message