The chart repeats the timing experiments for all benchmarks (and their variations) discussed in our OOPSLA 2004 paper. That paper contains detailed descriptions of each benchmark: in the table below, we have listed the relevant section numbers. In the column marked Java?, a "+" indicates that the program is in fact pure Java, while a "-" indicates the use of special features of AspectJ.
All experiments where run with three compilers:
The main purpose of the abc+soot combination is to determine how much of abc's speedup is due to standard optimisations that are not particular to AspectJ. It appears that on the whole the effect is minimal, except in the case of the LoD benchmark, where soot -O yields an improvement of 325%. Further investigation reveals this is due to the fact that ajc generates a very large number of locals, which Soot is able to eliminate.
For each compiler, we took three measurements:
The purpose of running with both JIT and interpreter is to factor out the effect of JIT optimisations. This is apparent in the ProdLin benchmark, where ajc outperforms abc on the JIT, but not on the interpreter. Naturally we aim for this not to happen, but it is notoriously difficult to predict the effects of JIT optimisations. The purpose of running with no inlining is to test the hypothesis, stated in Hilsdale's and Hugunin's paper on the internals of ajc (pdf), that AspectJ compilers can rely on the JIT for most inlining decisions. There seems to be little effect from turning off inlining, however, so there may be value in a AspectJ-specific inliner. abc does not yet have such an inliner.
The machine configuration was a dual Pentium 4 Xeon 3.2 Ghz with 4 GB of ram, running Debian sarge. We used the HotSpot JVM, build 1.4.2_03-b02 for all experiments except those involving *J, which require JVM 1.4.0-b92 due to the peculiarities of JVMPI.
Times are given in seconds. In order to obtain reliable numbers, each measurement was performed 14 times; the highest and lowest measurement were discarded, and then the average of the remaining 12 numbers was computed. Next to each time in seconds is a figure in parentheses; this is the speedup relevant to straight ajc. So for example, we divided the normal time of LoD cflow for ajc and abc to obtain a factor of 45.90.
Finally, we note that while abc makes substantial improvements over ajc, there is still ample of room for further optimization, comparing the pure Java programs to their AspectJ counterparts. We expect to report on these in the near future.
| benchmark | ajc | ajc+soot | abc | ||||||||||||||
| name | section | java? | normal | noinline | interpreter | normal | noinline | interpreter | normal | noinline | interpreter | ||||||
| dcm | 5.2.1 | - | 3.59 | 3.73 | 32.71 | 3.63 | (0.99) | 3.74 | (1.00) | 35.25 | (0.93) | 3.59 | (1.00) | 3.65 | (1.02) | 33.68 | (0.97) |
| dcm (base program) | 5.2.1 | + | 0.22 | 0.22 | 0.58 | 0.22 | (1.00) | 0.22 | (1.00) | 0.53 | (1.09) | 0.22 | (1.00) | 0.22 | (1.00) | 0.58 | (1.00) |
| prodLine | 5.2.2 | - | 0.95 | 0.97 | 7.66 | 0.95 | (0.99) | 0.99 | (0.98) | 7.73 | (0.99) | 1.62 | (0.58) | 1.62 | (0.60) | 7.49 | (1.02) |
| tetris | 5.2.3 | - | 119.36 | 119.35 | 119.36 | 119.35 | (1.00) | 119.35 | (1.00) | 119.35 | (1.00) | 119.35 | (1.00) | 119.34 | (1.00) | 119.35 | (1.00) |
| bean (original) | 5.2.4 | - | 1.31 | 1.35 | 4.79 | 1.31 | (1.00) | 1.31 | (1.03) | 4.75 | (1.01) | 1.31 | (1.00) | 1.37 | (0.99) | 4.77 | (1.00) |
| bean (improved) | 5.2.4 | - | 1.15 | 1.20 | 3.71 | 1.20 | (0.96) | 1.20 | (1.00) | 3.63 | (1.02) | 1.20 | (0.96) | 1.23 | (0.97) | 3.63 | (1.02) |
| bean (pure Java) | 5.2.4 | + | 1.15 | 1.15 | 3.15 | 1.15 | (1.00) | 1.15 | (1.00) | 3.18 | (0.99) | 1.15 | (1.00) | 1.17 | (0.98) | 3.17 | (0.99) |
| nullcheck (original) | 5.3.1 | - | 17.59 | 19.90 | 120.15 | 17.26 | (1.02) | 19.62 | (1.01) | 121.45 | (0.99) | 5.43 | (3.24) | 6.33 | (3.14) | 45.84 | (2.62) |
| nullcheck (improved) | 5.3.1 | - | 5.41 | 6.09 | 36.90 | 5.25 | (1.03) | 5.76 | (1.06) | 35.62 | (1.04) | 1.57 | (3.45) | 1.73 | (3.53) | 16.62 | (2.22) |
| nullcheck (not within) | 5.3.1 | - | 1.52 | 1.83 | 13.52 | 1.36 | (1.12) | 1.67 | (1.10) | 12.44 | (1.09) | 1.88 | (0.81) | 1.78 | (1.03) | 17.18 | (0.79) |
| nullcheck (after) | 5.3.1 | - | 1.41 | 1.62 | 10.67 | 1.57 | (0.90) | 1.81 | (0.90) | 11.14 | (0.96) | 1.41 | (1.00) | 1.62 | (1.00) | 11.54 | (0.93) |
| nullcheck (pure Java) | 5.3.1 | + | 1.20 | 1.31 | 8.58 | 1.15 | (1.04) | 1.26 | (1.04) | 9.70 | (0.88) | 1.15 | (1.04) | 1.20 | (1.09) | 8.98 | (0.96) |
| figure | 5.3.2 | - | 6.99 | 8.07 | 49.72 | 6.66 | (1.05) | 7.79 | (1.04) | 49.79 | (1.00) | 2.82 | (2.48) | 3.03 | (2.67) | 15.06 | (3.30) |
| figure (threadsafe count) | 5.3.2 | + | 0.53 | 0.58 | 4.01 | 0.53 | (1.00) | 0.58 | (1.00) | 4.04 | (0.99) | 0.53 | (1.00) | 0.58 | (1.00) | 3.98 | (1.01) |
| figure (non-safe count) | 5.3.2 | + | 0.22 | 0.27 | 2.48 | 0.22 | (0.98) | 0.27 | (1.00) | 2.35 | (1.06) | 0.22 | (0.98) | 0.27 | (1.00) | 2.31 | (1.07) |
| figure (pure Java) | 5.3.2 | + | 0.11 | 0.21 | 1.32 | 0.11 | (1.00) | 0.22 | (0.97) | 1.34 | (0.99) | 0.11 | (1.00) | 0.22 | (0.97) | 1.35 | (0.98) |
| LoD cflow | 5.3.3 | - | 45.90 | 47.80 | 116.59 | 14.11 | (3.25) | 15.45 | (3.09) | 91.04 | (1.28) | 1.00 | (45.90) | 1.00 | (47.80) | 3.35 | (34.84) |
| LoD (no cflow) | 5.3.3 | - | 0.63 | 0.58 | 1.58 | 0.61 | (1.04) | 0.58 | (1.00) | 1.56 | (1.01) | 0.63 | (1.00) | 0.58 | (1.00) | 1.45 | (1.09) |