Re: [abc] oopsla paper

After prolonged investigation and discussion with Ganesh on IRC (check
logs for details), we decided the following:

- The jumps between 4s and 28s are caused by java and the X server
running at the same priority, so that it is essentially random which
gets more CPU time -- and those two results correspond to the two cases.
- We do not want to measure the times for a 'minimised animation', since
Oege was rather clear he wanted "real world measurements" -- we were of
the opinion that a pure CPU time measurement would be unfair.
- We checked the priority the X server was running at on Oege's machine
where (we assume) the original measurements were made. Setting my own X
server to that priority eliminates the jumps (the result is
deterministically 28s).

In view of all this, I am now re-running all benchmarks with my own X
server set to Oege's priority. When I have these numbers, I will post
them here, and then run the tracematch version while collecting the
memory data needed for figure 6. Then I'll rewrite any text that needs
rewriting, and hopefully will have a final version of the paper in CVS
by later tonight.

- P

Pavel Avgustinov wrote:

>While I'm still rather at a loss for what to do, here the running times
>for the benchmark on my laptop:
>
>JHotDraw with minimised animation window:
>- Pure java: 5s
>- Naive aspects: 22s
>- Seminaive aspects: 11.2s
>- Non-naive aspects: 11.1s
>- Tracematches: between 220 and 256 s.
>
>Clearly it would be better to quote these numbers rather than the ones I
>obtained on my desktop, since we "only" have a slowdown of about 50...
>but the current numbers in the paper show a slowdown of less than 30.
>
>What should we do? Laurie? Oege?
>
>- P
>
>Pavel Avgustinov wrote:
>
>
>
>>Pavel Avgustinov wrote:
>>
>>
>>
>>
>>
>>>I plan to call it a day at some point this afternoon and replace the
>>>numbers by newly generated ones on my own machine. I'll check the
>>>section for any re-wording that needs to be done. There should be some
>>>improvement, though nothing dramatic -- we may be 15 times slower
>>>instead of 20, that order of magnitude.
>>>
>>>
>>>
>>>
>>>
>>>
>>Slight problem with that...
>>
>>When trying to measure how long the pure-java version of the benchmark
>>took, I was surprised to find it fluctuating, seemingly randomly, betwee
>>4.4 seconds and 28.2 seconds -- always these two values, with a margin
>>of error of 300ms or so. Investigation revealed that the running time
>>depended on whether the X server or the JVM was prioritised by the OS --
>>if the JVM got more CPU time, it finished in 4.4 seconds, if the X
>>server got more it was 28.2.
>>
>>I tried the following experiment: I minimised the MDI child window
>>containing the animation before selecting Animation->Start from the
>>menu. The results are somewhat catastrophic... The pure java version
>>finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
>>accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
>>seconds, with the expected slight upwards trend in memory usage for
>>seminaive and no such trend for non-naive aspects.
>>
>>Tracematches, on a minimised animation window, took me 194 seconds.
>>
>>We thus have a slowdown of almost a factor 400, when ignoring the X
>>server overheads for the animation.
>>
>>I'm a bit at a loss for what to do. I can't seem to get reliable numbers
>>for the pure java version with a non-minimised animation window, but the
>>numbers with minimised windows seem to destroy our 'happy story'. We
>>could leave the numbers as they are and investigate performance further
>>until OOPSLA... or try to obtain some sort of numbers with non-minimised
>>animation windows, these would be reasonably close to the ones in the
>>paper..
>>
>>Here a summary of the timings I've taken:
>>
>>JHotDraw, minimising MDI animation window before starting animation:
>>- pure Java: 0.5 s
>>- naive aspects: 13.5 s
>>- seminaive aspects: 7.3 s
>>- non-naive aspects: 7.3 s
>>- tracematches: 194 s
>>
>>JHotDraw, without minimising MDI animation window before starting
>>animation, keeping it visible at all times:
>>- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
>>continuously)
>>- tracematches: 300-315 s.
>>
>>Any suggestions on a course of action are very welcome. If you can run
>>the benchmark yourself, it might be helpful to compare results.
>>
>>- P
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>
>
Received on Thu Aug 11 20:37:57 2005