Re: [abc] oopsla paper

I have now checked in the numbers into CVS, as well as (most of) my
changes to the abc code -- the optimisations could do with a large
amount of cleaning up, most of my time was spent trying different things
rather than beautifying code. ;-)

The last thing that remains to do is change figure 6, which doesn't
reflect the current reality. Please take a look at
http://musketeer.comlab.ox.ac.uk/~pavel/memusage.pdf -- there are two
graphs of memory usage there, taken at 5 second intervals. The
difference is that for the top graph, I called System.gc() before
reporting memory usage every five seconds, while for the bottom graph no
such call was made. Since System.gc() tends to collect all junk objects,
the top curve is rather smoother. The bottom one looks more like what we
have currently in the paper. Which one should I include? The text seems
to suggest that we call the GC before each measurement, which would
indicate the top curve. The much more regular memory behaviour could be
explained with the optimisations I've introduced -- I have eliminated
99.95% of disjunct object allocations that were present when the current
figure 6 was generated.

Also comments on the style of the graphs are welcome -- are jagged lines
better than smooth ones, say, or should I use a different colour.

Please comment on these issues tonight or tomorrow morning, as I'd like
to put the final version of the figure into the paper around lunch-time
and submit in the early afternoon.

- P

Prof. Laurie HENDREN wrote:

>Those numbers look fine.
>
>Laurie
>
>
>+-------------------------------------------------------------+
>| Laurie Hendren, Professor, School of Computer Science |
>| McGill University |
>| 318 McConnell Engineering Building tel: (514) 398-7391 |
>| 3480 University Street fax: (514) 398-3883 |
>| Montreal, Quebec H3A 2A7 hendren@cs.mcgill.ca |
>| CANADA http://www.sable.mcgill.ca/~hendren |
>| http://wwww.sable.mcgill.ca http://aspectbench.org |
>+-------------------------------------------------------------+
>
>On Thu, 11 Aug 2005, Pavel Avgustinov wrote:
>
>
>
>>Here the numbers that will go into the final paper:
>>
>>- Pure java: 28.8s, 1,092K
>>- Naive aspect: 49.3s, 78,682K (I'm not sure why it suddenly takes more
>>memory... I can only blame it on the JVM -- this is Sun's Java5 rather
>>than Hotspot)
>>- Weakref aspect: 36.8s, 1,413K
>>- Tracematches: 275.2s, 1,103K
>>
>>I'm about to run the tracematch version, dumping memory usage every 5
>>seconds (for figure 6).
>>
>>- P
>>
>>Pavel Avgustinov wrote:
>>
>>
>>
>>>After prolonged investigation and discussion with Ganesh on IRC (check
>>>logs for details), we decided the following:
>>>
>>>- The jumps between 4s and 28s are caused by java and the X server
>>>running at the same priority, so that it is essentially random which
>>>gets more CPU time -- and those two results correspond to the two cases.
>>>- We do not want to measure the times for a 'minimised animation', since
>>>Oege was rather clear he wanted "real world measurements" -- we were of
>>>the opinion that a pure CPU time measurement would be unfair.
>>>- We checked the priority the X server was running at on Oege's machine
>>>where (we assume) the original measurements were made. Setting my own X
>>>server to that priority eliminates the jumps (the result is
>>>deterministically 28s).
>>>
>>>In view of all this, I am now re-running all benchmarks with my own X
>>>server set to Oege's priority. When I have these numbers, I will post
>>>them here, and then run the tracematch version while collecting the
>>>memory data needed for figure 6. Then I'll rewrite any text that needs
>>>rewriting, and hopefully will have a final version of the paper in CVS
>>>by later tonight.
>>>
>>>- P
>>>
>>>Pavel Avgustinov wrote:
>>>
>>>
>>>
>>>
>>>
>>>>While I'm still rather at a loss for what to do, here the running times
>>>>for the benchmark on my laptop:
>>>>
>>>>JHotDraw with minimised animation window:
>>>>- Pure java: 5s
>>>>- Naive aspects: 22s
>>>>- Seminaive aspects: 11.2s
>>>>- Non-naive aspects: 11.1s
>>>>- Tracematches: between 220 and 256 s.
>>>>
>>>>Clearly it would be better to quote these numbers rather than the ones I
>>>>obtained on my desktop, since we "only" have a slowdown of about 50...
>>>>but the current numbers in the paper show a slowdown of less than 30.
>>>>
>>>>What should we do? Laurie? Oege?
>>>>
>>>>- P
>>>>
>>>>Pavel Avgustinov wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>Pavel Avgustinov wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>I plan to call it a day at some point this afternoon and replace the
>>>>>>numbers by newly generated ones on my own machine. I'll check the
>>>>>>section for any re-wording that needs to be done. There should be some
>>>>>>improvement, though nothing dramatic -- we may be 15 times slower
>>>>>>instead of 20, that order of magnitude.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>Slight problem with that...
>>>>>
>>>>>When trying to measure how long the pure-java version of the benchmark
>>>>>took, I was surprised to find it fluctuating, seemingly randomly, betwee
>>>>>4.4 seconds and 28.2 seconds -- always these two values, with a margin
>>>>>of error of 300ms or so. Investigation revealed that the running time
>>>>>depended on whether the X server or the JVM was prioritised by the OS --
>>>>>if the JVM got more CPU time, it finished in 4.4 seconds, if the X
>>>>>server got more it was 28.2.
>>>>>
>>>>>I tried the following experiment: I minimised the MDI child window
>>>>>containing the animation before selecting Animation->Start from the
>>>>>menu. The results are somewhat catastrophic... The pure java version
>>>>>finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
>>>>>accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
>>>>>seconds, with the expected slight upwards trend in memory usage for
>>>>>seminaive and no such trend for non-naive aspects.
>>>>>
>>>>>Tracematches, on a minimised animation window, took me 194 seconds.
>>>>>
>>>>>We thus have a slowdown of almost a factor 400, when ignoring the X
>>>>>server overheads for the animation.
>>>>>
>>>>>I'm a bit at a loss for what to do. I can't seem to get reliable numbers
>>>>>for the pure java version with a non-minimised animation window, but the
>>>>>numbers with minimised windows seem to destroy our 'happy story'. We
>>>>>could leave the numbers as they are and investigate performance further
>>>>>until OOPSLA... or try to obtain some sort of numbers with non-minimised
>>>>>animation windows, these would be reasonably close to the ones in the
>>>>>paper..
>>>>>
>>>>>Here a summary of the timings I've taken:
>>>>>
>>>>>JHotDraw, minimising MDI animation window before starting animation:
>>>>>- pure Java: 0.5 s
>>>>>- naive aspects: 13.5 s
>>>>>- seminaive aspects: 7.3 s
>>>>>- non-naive aspects: 7.3 s
>>>>>- tracematches: 194 s
>>>>>
>>>>>JHotDraw, without minimising MDI animation window before starting
>>>>>animation, keeping it visible at all times:
>>>>>- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
>>>>>continuously)
>>>>>- tracematches: 300-315 s.
>>>>>
>>>>>Any suggestions on a course of action are very welcome. If you can run
>>>>>the benchmark yourself, it might be helpful to compare results.
>>>>>
>>>>>- P
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
>
>
Received on Thu Aug 11 23:05:39 2005