On 21/06/2011 07:01 PM, Warp wrote:

>    I have always detested interlanguage benchmark comparisons (even in
> situations where my pet language comes on top). The reason is that making
> a fair comparison is extremely difficult and what can be considered "fair"
> is very up to interpretation.

>    It may be that the best way to implement the task is different in the
> languages, eg. because of technical reasons. If the aim is
> to implement a task in the most efficient way in both languages, these
> implementations may often need to use quite different approaches.
>
>    Another possible approach is to take an expert, competent programmer for
> each of the languages and tell them to implement the task as efficiently
> as they can.
>
>    This can be a much fairer approach, but might suffer from the
> implementations being detached from actual, practical programs and might
> in fact not represent the *typical* (but still competent) average solution
> in that language. In other words, if this becomes a competition on who comes
> up with the fastest solution, the end result might be a mess of hacker
> optimization that has little to do with practical programming. It might be
> fast, but it usually won't be very good code (in terms of maintainability,
> readability, etc.)

I sometimes joke that The Great Language Shootout is a benchmark to 
measure how many domain experts there are for each programming language, 
and how many hours of spare time they have.

Unfortunately, run-time and memory usage are easy to measure, while 
"readability", "maintainability" and similar concepts are almost 
infeasible to measure.

The series of benchmarks I ran a while back was an attempt to see how 
fast C++ and Haskell are if you attempt to solve problems in the 
"typical" way for each language. I mean, let's face it, if some chunk of 
your program is utterly performance-critical, there's always assembly 
language. Maintainable? Not really.

Trouble is, "typical" isn't very well-defined either...

Post a reply to this message