Re: [abc] oopsla paper

Here the numbers that will go into the final paper:

- Pure java: 28.8s, 1,092K
- Naive aspect: 49.3s, 78,682K (I'm not sure why it suddenly takes more
memory... I can only blame it on the JVM -- this is Sun's Java5 rather
than Hotspot)
- Weakref aspect: 36.8s, 1,413K
- Tracematches: 275.2s, 1,103K

I'm about to run the tracematch version, dumping memory usage every 5
seconds (for figure 6).

- P

Pavel Avgustinov wrote:

>After prolonged investigation and discussion with Ganesh on IRC (check
>logs for details), we decided the following:
>
>- The jumps between 4s and 28s are caused by java and the X server
>running at the same priority, so that it is essentially random which
>gets more CPU time -- and those two results correspond to the two cases.
>- We do not want to measure the times for a 'minimised animation', since
>Oege was rather clear he wanted "real world measurements" -- we were of
>the opinion that a pure CPU time measurement would be unfair.
>- We checked the priority the X server was running at on Oege's machine
>where (we assume) the original measurements were made. Setting my own X
>server to that priority eliminates the jumps (the result is
>deterministically 28s).
>
>In view of all this, I am now re-running all benchmarks with my own X
>server set to Oege's priority. When I have these numbers, I will post
>them here, and then run the tracematch version while collecting the
>memory data needed for figure 6. Then I'll rewrite any text that needs
>rewriting, and hopefully will have a final version of the paper in CVS
>by later tonight.
>
>- P
>
>Pavel Avgustinov wrote:
>
>
>
>>While I'm still rather at a loss for what to do, here the running times
>>for the benchmark on my laptop:
>>
>>JHotDraw with minimised animation window:
>>- Pure java: 5s
>>- Naive aspects: 22s
>>- Seminaive aspects: 11.2s
>>- Non-naive aspects: 11.1s
>>- Tracematches: between 220 and 256 s.
>>
>>Clearly it would be better to quote these numbers rather than the ones I
>>obtained on my desktop, since we "only" have a slowdown of about 50...
>>but the current numbers in the paper show a slowdown of less than 30.
>>
>>What should we do? Laurie? Oege?
>>
>>- P
>>
>>Pavel Avgustinov wrote:
>>
>>
>>
>>
>>
>>>Pavel Avgustinov wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>I plan to call it a day at some point this afternoon and replace the
>>>>numbers by newly generated ones on my own machine. I'll check the
>>>>section for any re-wording that needs to be done. There should be some
>>>>improvement, though nothing dramatic -- we may be 15 times slower
>>>>instead of 20, that order of magnitude.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>Slight problem with that...
>>>
>>>When trying to measure how long the pure-java version of the benchmark
>>>took, I was surprised to find it fluctuating, seemingly randomly, betwee
>>>4.4 seconds and 28.2 seconds -- always these two values, with a margin
>>>of error of 300ms or so. Investigation revealed that the running time
>>>depended on whether the X server or the JVM was prioritised by the OS --
>>>if the JVM got more CPU time, it finished in 4.4 seconds, if the X
>>>server got more it was 28.2.
>>>
>>>I tried the following experiment: I minimised the MDI child window
>>>containing the animation before selecting Animation->Start from the
>>>menu. The results are somewhat catastrophic... The pure java version
>>>finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
>>>accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
>>>seconds, with the expected slight upwards trend in memory usage for
>>>seminaive and no such trend for non-naive aspects.
>>>
>>>Tracematches, on a minimised animation window, took me 194 seconds.
>>>
>>>We thus have a slowdown of almost a factor 400, when ignoring the X
>>>server overheads for the animation.
>>>
>>>I'm a bit at a loss for what to do. I can't seem to get reliable numbers
>>>for the pure java version with a non-minimised animation window, but the
>>>numbers with minimised windows seem to destroy our 'happy story'. We
>>>could leave the numbers as they are and investigate performance further
>>>until OOPSLA... or try to obtain some sort of numbers with non-minimised
>>>animation windows, these would be reasonably close to the ones in the
>>>paper..
>>>
>>>Here a summary of the timings I've taken:
>>>
>>>JHotDraw, minimising MDI animation window before starting animation:
>>>- pure Java: 0.5 s
>>>- naive aspects: 13.5 s
>>>- seminaive aspects: 7.3 s
>>>- non-naive aspects: 7.3 s
>>>- tracematches: 194 s
>>>
>>>JHotDraw, without minimising MDI animation window before starting
>>>animation, keeping it visible at all times:
>>>- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
>>>continuously)
>>>- tracematches: 300-315 s.
>>>
>>>Any suggestions on a course of action are very welcome. If you can run
>>>the benchmark yourself, it might be helpful to compare results.
>>>
>>>- P
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>
>
Received on Thu Aug 11 21:44:53 2005