http://www.codeproject.com/KB/cross-platform/BenchmarkCppVsDotNet.aspx

It should be noted that the compact framework doesn't do nearly as much work 
optimizing stuff that the desktop systems do, and that's well known. The GC 
is known to suck, if nothing else, but overall a lot of stuff is interpreted 
without caching and such rather than use all the tricks that go into modern 
VM-based systems.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> http://www.codeproject.com/KB/cross-platform/BenchmarkCppVsDotNet.aspx

> It should be noted that the compact framework doesn't do nearly as much work 
> optimizing stuff that the desktop systems do, and that's well known. The GC 
> is known to suck, if nothing else, but overall a lot of stuff is interpreted 
> without caching and such rather than use all the tricks that go into modern 
> VM-based systems.

  I have always detested interlanguage benchmark comparisons (even in
situations where my pet language comes on top). The reason is that making
a fair comparison is extremely difficult and what can be considered "fair"
is very up to interpretation.

  How do you implement the same program in two different programming
languages so that it's fair for both?

  One solution would be to make both implementations as identical as
possible to each other, by always using the closest thing in both languages
in every situation. However, this can be very unfair because it very often
happens that this solution favors one of the languages over the other,
introducing a bad bias.

  It may be that the best way to implement the task is different in the
languages, eg. because of technical reasons. For example, memory allocation
sucks in terms of speed in most C++ compilers, while in many other languages
it's very fast. OTOH, C++ often beats those other languages in raw number
crunching that does not involve constant memory allocation. If the aim is
to implement a task in the most efficient way in both languages, these
implementations may often need to use quite different approaches.

  Another possible approach is to take an expert, competent programmer for
each of the languages and tell them to implement the task as efficiently
as they can.

  This can be a much fairer approach, but might suffer from the
implementations being detached from actual, practical programs and might
in fact not represent the *typical* (but still competent) average solution
in that language. In other words, if this becomes a competition on who comes
up with the fastest solution, the end result might be a mess of hacker
optimization that has little to do with practical programming. It might be
fast, but it usually won't be very good code (in terms of maintainability,
readability, etc.)

-- 
                                                          - Warp

Post a reply to this message

On 6/21/2011 11:01, Warp wrote:
>    I have always detested interlanguage benchmark comparisons (even in
> situations where my pet language comes on top). The reason is that making
> a fair comparison is extremely difficult and what can be considered "fair"
> is very up to interpretation.

Agreed. That's why I thought all the comments on the article about how it 
was more a comparison of the standard libraries used were missing the point 
of what he was benchmarking. Clearly there's a difference between "the 
fastest you can get out of a particular language/compiler/environment 
regardless of the cost of the implementation" and "the speed it goes when 
you do everything normally."

Clearly his measuring of P/Invoke overhead shows he understands how these 
things work.

But it *is* interesting on a general level. I've heard "Ruby is slow" and 
I've heard "Python is slow", but I haven't seen any hard numbers at all 
there. How slow? 10x as slow? 100x as slow? Etc. Saying "you'll need 20% 
more servers in order to finish the project in 35% less time" might be a 
good trade-off, might be a bad trade-off. But without even a vague idea, 
it's impossible to say.

The problem is that the vocal people who discuss these things in public are 
generally people who are trying to prove a point, so it's never treated as a 
value-per-unit-effort comparison and instead is always treated as "maybe my 
jet flies into a mountain if you do a garbage-collect at the wrong time, oh 
noes!"

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 21/06/2011 07:01 PM, Warp wrote:

>    I have always detested interlanguage benchmark comparisons (even in
> situations where my pet language comes on top). The reason is that making
> a fair comparison is extremely difficult and what can be considered "fair"
> is very up to interpretation.

>    It may be that the best way to implement the task is different in the
> languages, eg. because of technical reasons. If the aim is
> to implement a task in the most efficient way in both languages, these
> implementations may often need to use quite different approaches.
>
>    Another possible approach is to take an expert, competent programmer for
> each of the languages and tell them to implement the task as efficiently
> as they can.
>
>    This can be a much fairer approach, but might suffer from the
> implementations being detached from actual, practical programs and might
> in fact not represent the *typical* (but still competent) average solution
> in that language. In other words, if this becomes a competition on who comes
> up with the fastest solution, the end result might be a mess of hacker
> optimization that has little to do with practical programming. It might be
> fast, but it usually won't be very good code (in terms of maintainability,
> readability, etc.)

I sometimes joke that The Great Language Shootout is a benchmark to 
measure how many domain experts there are for each programming language, 
and how many hours of spare time they have.

Unfortunately, run-time and memory usage are easy to measure, while 
"readability", "maintainability" and similar concepts are almost 
infeasible to measure.

The series of benchmarks I ran a while back was an attempt to see how 
fast C++ and Haskell are if you attempt to solve problems in the 
"typical" way for each language. I mean, let's face it, if some chunk of 
your program is utterly performance-critical, there's always assembly 
language. Maintainable? Not really.

Trouble is, "typical" isn't very well-defined either...

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> Trouble is, "typical" isn't very well-defined either...

  "Typical" also depends on the language. What is typical in one language
might not be in another. (For example, in Java it's normal and expected for
the code to be full of 'new' keywords everywhere. If you do that in C++,
most expert C++ programmers will cringe.)

-- 
                                                          - Warp

Post a reply to this message

On 6/22/2011 10:44, Warp wrote:
> Invisible<voi### [at] devnull>  wrote:
>> Trouble is, "typical" isn't very well-defined either...
>
>    "Typical" also depends on the language. What is typical in one language
> might not be in another. (For example, in Java it's normal and expected for
> the code to be full of 'new' keywords everywhere. If you do that in C++,
> most expert C++ programmers will cringe.)
>

Sure. And on the third hand, what's typical in a language gets optimized in 
that language. It's typical for virtual functions to not go through a 
dispatch table in Java if they don't actually have any other class 
overriding them, while C++ can't generally manage this. So it's "typical" to 
not make things virtual in C++ that are virtual in Java, but Java optimizes 
that bit and C++ doesn't.

For example, it's also "typical" (or at least getting typical) for the Java 
compiler to optimize out heap allocations (i.e., turn "new" into an alloca() 
sort of operation) in ways that C++ would never try.

That's why it's useful to benchmark "typical" ways of doing things. Anything 
small enough to not be different in "typical" methods is probably not 
something that's useful to benchmark, unless you're (say) comparing two tiny 
programs in the *same* language to see which VM or compiler does a better 
optimization job.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> Sure. And on the third hand, what's typical in a language gets optimized in 
> that language. It's typical for virtual functions to not go through a 
> dispatch table in Java if they don't actually have any other class 
> overriding them, while C++ can't generally manage this. So it's "typical" to 
> not make things virtual in C++ that are virtual in Java, but Java optimizes 
> that bit and C++ doesn't.

  OTOH the slowness of virtual function calls (vs. regular function calls)
is often greatly exaggerated, based solely on presuppositions rather than
actual mesurements.

> For example, it's also "typical" (or at least getting typical) for the Java 
> compiler to optimize out heap allocations (i.e., turn "new" into an alloca() 
> sort of operation) in ways that C++ would never try.

  I wonder when they will add scope-bound objects to Java.

  First Java wanted to get as far from C++ as possible. It's slowly drifting
towards it.

-- 
                                                          - Warp

Post a reply to this message

On 6/22/2011 12:28, Warp wrote:
>    OTOH the slowness of virtual function calls (vs. regular function calls)
> is often greatly exaggerated, based solely on presuppositions rather than
> actual mesurements.

Well, it also depends on the programming language. In a language like C++ or 
Eiffel, you can make it a constant factor slower than a normal dispatch. In 
a language like Python or Smalltalk, where executing the third line of the 
function can change whether the fourth line of the function is a virtual 
dispatch or not, it's much harder to optimize. I suspect the exaggerated 
slowness claims is left over from when "virtual dispatch" meant "look up a 
string in a hash table for each function call" rather than "index into a 
vtable".

>    I wonder when they will add scope-bound objects to Java.

They did. They called it C#. ;-)   (Altho, unlike C++, it's actually 
"instances of scope-based classes" rather than "scope-based objects." And 
even there, that's just the implementation and not necessarily the 
semantics. There's nothing stopping you from allocating value types on the 
heap in a C# implementation.)

My guess is, it'll take a very long time, because unlike C#, people are 
unwilling to change the .class file format. The same reason C# has real 
generics and Java has type-erased generics.

>    First Java wanted to get as far from C++ as possible. It's slowly drifting
> towards it.

Nah. Only performance-wise. These are all performance-oriented things done 
by the JVM after seeing the class file. It's not changing the syntax or 
semantics of Java at all (unlike scope-based objects would).

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 22/06/2011 08:05 PM, Darren New wrote:

> Sure. And on the third hand, what's typical in a language gets optimized
> in that language.

> That's why it's useful to benchmark "typical" ways of doing things.

What he just said.

I'm sure most compiled languages can be made to go really fast, if you 
throw enough effort at the problem. A much more useful question is "how 
easy is it?"

I didn't get far with my benchmarking attempt. About the only thing I 
managed to prove is "C++ makes it easy to write reasonably fast code. 
Haskell makes it jaw-droppingly easy to accidentally write horrifyingly 
slow code." Useful to know.

Post a reply to this message