Moby Disk Consulting
Software Development, Training & Consulting
William Garrison - mobydisk at mobydisk daht com

JIT -vs- Compilers

Do just-in-time (JIT) compiled languages really hold-up against fully-compiled code? Here is a very quick benchmark to try to answer this question.

Why?

I've been interested in compiler performance ever since I wrote my first video game. JIT compilers provide more features than their fully-compiled counterparts, and claim comparable performance. In theory, JIT compilers have the potential to outpace compiled languages. Today, that is not the case at all. But how close are we? What is the performance penalty for using C# over C++?

Benchmark Design

This particular test attempts to perform an apples-to-apples comparison. The benchmark is integer and floating-point math, with lots of opportunities for optimizations such as inlining and loop unrolling. No strings, collections, or high-level structures are used. No memory allocations take place during the main loop. No garbage collection applies. These simplicities eliminate the possibility of arguing over implementations of the various APIs, Frameworks, and memory architectures.

The Code

The code performs my variant of the Buffon needle experiment, which is a simple way to compute the value of pi. My implementation uses points, not lines so it is simpler and less efficient than the original experiment. This means the main loop is only a few lines long. The code relies on a random number generator, so a simple 64-bit linear congruent pseudo-random generator is included. It will produce the same sequence of numbers in each language, and the entire generator could be inlined if the compiler chooses to do so.

For a quick peek, you can view the C++ source code or the C# source code. This source code ZIP file contains the source code and project files for MSVC 6.0, MSVC 2003, MSVC 2005, C# 1.1, and C# 2.0.

The Results

The results appear in the table below. Higher scores are better:

CompilerConfigurationResult
C# 1.1 Release
0.65
C# 2.0 Release
0.7
MSVC 6 Release
0.77
MSVC 6 Optimized
0.97
MSVC 2005 Release
1.2
MSVC 2003 Release
1.36
MSVC 2003 Optimized
1.4
MSVC 2005 Optimized
1.45

The unit is the number of tests completed per second. Each test is 10 million iterations of the algorithm. The test was run on a P4 2.4GHz running Windows XP.

For C# and C++, "release" refers to whatever the default release configuration is. "Optimized" refers to the optimal performance settings. For C++, this includes floating-point consistency optimizations that may run the FPU outside of the IEEE 754 specification.

Conclusion

It looks like JIT comes in at about half the speed of a purely compiled language. For desktop applications this is probably just fine. For high-performance applications this may be unsuitable. JIT compilers aren't yet performing faster than native compiled languages, but they aren't too far behind.

I was surprised by the variance in the versions of Microsoft Visual C++. Each new version is faster than the previous. The exception is that the 2003 edition was faster than 2005 in the default configuration, probably due to the new buffer security checks that are enabled in 2005. Much of the performance benefits may have to do with targeting newer CPUs: MSVC 6.0 targeted a 486/Pentium blend.

Future

I look forward to running this test against gcc, VB6, Mono, and Java just for fun.