Re: [abc] oopsla paper

Pavel Avgustinov wrote:

>I plan to call it a day at some point this afternoon and replace the
>numbers by newly generated ones on my own machine. I'll check the
>section for any re-wording that needs to be done. There should be some
>improvement, though nothing dramatic -- we may be 15 times slower
>instead of 20, that order of magnitude.
>
>
Slight problem with that...

When trying to measure how long the pure-java version of the benchmark
took, I was surprised to find it fluctuating, seemingly randomly, betwee
4.4 seconds and 28.2 seconds -- always these two values, with a margin
of error of 300ms or so. Investigation revealed that the running time
depended on whether the X server or the JVM was prioritised by the OS --
if the JVM got more CPU time, it finished in 4.4 seconds, if the X
server got more it was 28.2.

I tried the following experiment: I minimised the MDI child window
containing the animation before selecting Animation->Start from the
menu. The results are somewhat catastrophic... The pure java version
finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
seconds, with the expected slight upwards trend in memory usage for
seminaive and no such trend for non-naive aspects.

Tracematches, on a minimised animation window, took me 194 seconds.

We thus have a slowdown of almost a factor 400, when ignoring the X
server overheads for the animation.

I'm a bit at a loss for what to do. I can't seem to get reliable numbers
for the pure java version with a non-minimised animation window, but the
numbers with minimised windows seem to destroy our 'happy story'. We
could leave the numbers as they are and investigate performance further
until OOPSLA... or try to obtain some sort of numbers with non-minimised
animation windows, these would be reasonably close to the ones in the
paper..

Here a summary of the timings I've taken:

JHotDraw, minimising MDI animation window before starting animation:
- pure Java: 0.5 s
- naive aspects: 13.5 s
- seminaive aspects: 7.3 s
- non-naive aspects: 7.3 s
- tracematches: 194 s

JHotDraw, without minimising MDI animation window before starting
animation, keeping it visible at all times:
- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
continuously)
- tracematches: 300-315 s.

Any suggestions on a course of action are very welcome. If you can run
the benchmark yourself, it might be helpful to compare results.

- P
Received on Thu Aug 11 17:13:24 2005