CL Implementation Benchmarks on Win2000
Wednesday, June 30, 2004
The
cl-bench benchmarks that Eric Marsden has assembled are probably
the most comprehensive set of CL performance benchmarks
publicly available. They include the
Gabriel benchmarks, mathematical functions, bignum-intensive
operations, CLOS test, hashtable exercising, read-line exercising, and
various operations on arrays, strings and bitvectors. Rainer Joswig
has compiled a
set of results for different CL implementations on Mac OS X and
there is an equivalent
set of results for Linux/Unix CL implementations on
CLiki. However, I don't believe anyone has produced a similar
comparison for MS Windows CL implementations. Hence, this posting!
:-)
I ran tests against the following three CL implementations:
- Xanalys Lispworks version 4.3.7
- Franz Allegro CL version 6.2
- GNU CLISP version 2.33
All tests were run on a Toshiba Tecra 9000 laptop with a 1200MHz Intel Pentium III processor and 1 MB of RAM under MS Windows 2000 SP4.
Some caveats:
- "There are lies, damn lies, and benchmarks." ;-)
- Take these results with a grain of salt. They were not run under a strictly controlled environment (although they were run under approximately the same conditions).
- There has been no attempt to "weight" different tests. Such weightings would depend on the particular application that you intend to use the CL implementation for.
- Green indicates the best result.
- Orange indicates second place.
- Red indicates the worst result.
- "n/a" indicates that the test did not run successfully.
| Benchmark | LW-4.3.7 | ACL-6.2 | CLISP-2.33 |
| COMPILER | 3.28 | 2.48 | 2.28 |
| LOAD-FASL | 4.35 | 0.65 | 1.20 |
| SUM-PERMUTATIONS | 15.45 | 5.02 | 7.41 |
| WALK-LIST/SEQ | 0.21 | 0.25 | 0.09 |
| WALK-LIST/MESS | n/a | n/a | 0.03 |
| BOYER | 88.49 | 23.00 | 91.50 |
| BROWSE | 7.63 | 0.79 | 2.60 |
| DDERIV | 1.75 | 0.91 | 3.31 |
| DERIV | 1.98 | 0.88 | 3.46 |
| DESTRUCTIVE | 1.18 | 0.84 | 4.02 |
| DIV2-TEST-1 | 2.70 | 1.09 | 4.73 |
| DIV2-TEST-2 | 3.07 | 1.43 | 7.04 |
| FFT | 7.59 | 6.70 | 7.70 |
| FRPOLY/FIXNUM | 2.00 | 1.61 | 9.70 |
| FRPOLY/BIGNUM | 2.07 | 1.86 | 3.24 |
| FRPOLY/FLOAT | 2.50 | 2.63 | 8.98 |
| PUZZLE | 5.36 | 10.03 | 21.41 |
| TAK | 0.77 | 0.73 | 9.63 |
| CTAK | 0.78 | 1.90 | 7.55 |
| TRTAK | 0.77 | 0.72 | 9.27 |
| TAKL | 1.90 | 2.08 | 12.35 |
| STAK | 1.07 | 5.25 | 7.86 |
| FPRINT/UGLY | 5.99 | 9.82 | 5.07 |
| FPRINT/PRETTY | 23.73 | 11.06 | 4.31 |
| TRAVERSE | 6.58 | 3.51 | 22.74 |
| TRIANGLE | 3.92 | 8.61 | 36.20 |
| RICHARDS | 5.09 | 9.21 | 24.58 |
| FACTORIAL | 2.48 | 2.22 | 1.30 |
| FIB | 0.39 | 0.29 | 4.73 |
| FIB-RATIO | 0.18 | 9.21 | 0.07 |
| MANDELBROT/COMPLEX | 25.32 | 31.09 | 33.54 |
| MANDELBROT/DFLOAT | 13.11 | 15.09 | 30.92 |
| MRG32K3A | 13.84 | 7.79 | 42.78 |
| CRC40 | 90.64 | 86.25 | 161.34 |
| BIGNUM/ELEM-100-1000 | 0.74 | 1.62 | 0.09 |
| BIGNUM/ELEM-1000-100 | 2.87 | 8.02 | 0.27 |
| BIGNUM/ELEM-10000-1 | 3.73 | 22.42 | 0.25 |
| BIGNUM/PARI-100-10 | 0.08 | 0.12 | 0.03 |
| BIGNUM/PARI-200-5 | 0.38 | 0.57 | 0.10 |
| PI-DECIMAL/SMALL | 37.61 | 757.83 | 2.56 |
| PI-DECIMAL/BIG | n/a | 1727.69 | 1.88 |
| PI-ATAN | n/a | 11.97 | 2.10 |
| PI-RATIOS | 11.15 | 59.17 | 1.19 |
| SLURP-LINES | 0.01 | 0.02 | 0.02 |
| HASH-STRINGS | 2.05 | 14.35 | 3.05 |
| HASH-INTEGERS | 5.04 | 6.73 | 3.20 |
| BOEHM-GC | 11.94 | 9.29 | 25.83 |
| DEFLATE-FILE | n/a | 3.57 | 7.83 |
| 1D-ARRAYS | 0.69 | 0.51 | 1.49 |
| 2D-ARRAYS | 24.04 | 23.33 | 39.52 |
| 3D-ARRAYS | n/a | 48.18 | 87.82 |
| BITVECTORS | 2.83 | 3.20 | 16.92 |
| BENCH-STRINGS | 25.44 | 27.99 | 3.55 |
| fill-strings/adjustable | 66.62 | 97.89 | 64.58 |
| STRING-CONCAT | n/a | n/a | 564.81 |
| SEARCH-SEQUENCE | 7.48 | 7.47 | 18.52 |
| CLOS/defclass | 6.38 | 0.31 | 0.89 |
| CLOS/defmethod | 1.40 | 0.35 | 0.20 |
| CLOS/instantiate | 5.92 | 5.11 | 6.53 |
| CLOS/simple-instantiate | 1.26 | 0.88 | 6.08 |
| CLOS/methodcalls | 8.13 | 7.34 | 9.76 |
| CLOS/method+after | 6.67 | 3.11 | 8.07 |
| CLOS/complex-methods | 0.86 | 0.76 | n/a |
| EQL-SPECIALIZED-FIB | 1.06 | 0.96 | 3.12 |
To summarize:
- Lispworks was 1st place in 11 tests, 2nd place in 39 tests 3rd place in 8 tests and didn't complete 6 tests.
- Allegro CL was 1st place in 32 tests, 2nd place in 16 tests 3rd place in 14 tests and didn't complete 2 tests.
- CLISP was 1st place in 21 tests, 2nd place in 8 tests 3rd place in 34 tests and didn't complete 1 test.
