Clementson's Blog

ACL 7.0 is smokin!

Tuesday, November 9, 2004

Several months ago, I ran Eric Marsden's cl-bench benchmarks against the 3 most popular Win32 CL implemementations. Since ACL 7.0 is now available, I decided to see how it compares. Since I wanted to use the previous release as a a basis for comparison against the new release, I have kept the ACL 6.2 figures in the table. As before, all tests were run on a Toshiba Tecra 9000 laptop with a 1200MHz Intel Pentium III processor and 1 MB of RAM under MS Windows 2000 SP4. The color coding is as follows:

• Dark green indicates the best result.
• Light green indicates second place.
• Orange indicates third place.
• Red indicates the worst result.
• "n/a" indicates that the test did not run successfully.

"I mostly try to stay away from commenting on benchmarking except for making generalizations, because it is such a YMMV thing, and also because it is so transient; one platform emphasizes one thing and another emphasizes another; when one vendor finds a great algorithm the others scramble to try to leapfrog the first. So it is constantly changing.

To my knowledge, there is only one major functional reason why any of our lisp functionality should be slower than any others, and that is our 'wide-binding' symbol-access style - I've actually sped that up from 6.2 to 7.0, and am still looking at better ways to access symbol-value faster - you'll see speedups on any benchmarks that access global-special variables, and although you won't see any benchmarks to prove it, wide-binding allows lisp thread switches to become much faster, with both our 'virtual threads' and our 'os-threads' designs.

We have improved string/symbol hashing in two steps; in 6.2, we issued the hash.004 patch which improved the distribution of hash codes dramatically, and which reduced the collisions in these benchmarks to nearly zero. In 7.0, we also bashed things out in the symbol representation and thus increased the width of the hash-code generation field from 16 bits to 24 bits, which removed the rest of the collisions and spread the hash codes even more evenly. Your perception of snappiness is probably due to the two decimal orders of magnitude increase in hashing efficiency.

One final observation: The boyer benchmark seems much worse on 7.0, but I suspect that if you do a (room) on each of your 6.2 and 7.0 lisps, you'd find a much smaller newspace on whichever 7.0 image you are using to do the benchmarks. The boyer benchmark is cons-intensive, and depending on how many iterations you do the gc will figure into the benchmark speed. I don't know what the current thinking is on the inclusion of gc times, but back in the gabriel/stanford-benchmark times in the late '80s and easly '90s, I think that we had agreed that gc was undeterministic and thus should be factored out of benchmarks. This is borne out by the fact that even this set of benchmarks does a preliminary call to bench-gc, for each implementation to do with what they wish. I have not taken advantage of that, but in my own copy of sysdep/setup-acl.lisp, just before the (in-package :cl-bench), I placed this form:

(sys::resize-areas :new 15000000 :old 15000000)

which seems to help boyer out."

"Two things (perhaps it sounds too much like marketing hype to you, but it is truly how I, as a developer, do feel):

1. Results like this should serve as an answer to those who have guessed that we 'aren't interested' in speeding up particular kinds of benchmarks - obviously, as programmers explore newer or less-used areas of Lisp functionality, or as they come up with new benchmarks (especially those which showcase their favorite implementations' strong suits) there will always be some possibilities for temporary lags in performance as we weigh the requirements of our customers (especially paying ones). And, quite frankly, those areas might not always get immediate attention, since our highest priorities are in fact to satisfy requirements of our customers (or, more accurately, to do what we can to make them more successful). But putting these optimizations at lower priority doesn't necessarily mean that they won't get done; it just means that it won't get done as soon.
2. This isn't the end of it, by any means. I'm sure that other CL developers will continue (as they have) to improve the speed of their implementations, as will we. And the more meaningful benchmarks (both quantity and quality together) we see, the better CL implementations are likely to get."