[Soot-list] Class Loading - Bug or Feature?

[Steven Arzt]

Hi Marc-André,

I actually don't understand your approach. If you generate exclusions, you are loading less, not more, if you are using the -no-bodies-for-excluded option. What we did for FlowDroid was to stick to that option and manually forceResolve all classes on the classpath to HIERARCHY level. This took quite a while and consumed a lot of memory - for the Oracle JDK as well as for the Android SDK. It was something (out of the top of my mind, don't take this number too seriously) 700 seconds to load all the classes and quite some memory.

Best regards,
  Steven 

-----Ursprüngliche Nachricht-----
Von: soot-list-bounces at sable.mcgill.ca [mailto:soot-list-bounces at sable.mcgill.ca] Im Auftrag von Marc-André Laverdière
Gesendet: Mittwoch, 19. März 2014 20:18
An: soot-list at sable.mcgill.ca
Betreff: Re: [Soot-list] Class Loading - Bug or Feature?

Hi Steven,

I am not able to reproduce what you are saying. I am aggressively generating exclusions and keeping the libraries in the library scope.
The difference in memory is less than a meg when using a stubbed JDK jar.

When you did your experiments, were you using exclusions (with -no-bodies-for-excluded)? Or did you put them all in the application scope?

Marc-André Laverdière-Papineau
Doctorant - PhD Candidate

On 03/18/2014 05:09 PM, Steven Arzt wrote:
> Hi all,
> 
> This is expected behavior if you ask me. Soot only loads the classes it really really needs to load because class loading is expensive both in terms of memory and performance.
> 
> In your example, you are suggesting to load each and every class on the classpath to at least Hierarchy level because it might be a transitive descendant / implementer of List. We have experimented with such eager loading in FlowDroid because we experimented with algorithms that ask very similar questions as what you have in the example. After loading the basic classes, you can just iterate over the rest (dangling classes) and force-load them, so there is no need to have such a feature directly in Soot. Having said that, we dropped the idea altogether in our experiments because loading a compete hierarchy of the jdk is only feasible on big machines with lots of memory and takes forever. With Soot's current class loading and representation in memory, you want to avoid this at all costs.
> 
> I know that this is not the answer you're looking for and it was a pity since we had a really cool idea with FlowDroid (on-demand icfg), but apparently that's how it is.
> 
> Best regards,
> Steven
> 
> Marc-André Laverdière	 <marc-andre.laverdiere-papineau at polymtl.ca> wrote:
> 
>> Hello everybody,
>>
>> There are things about Soot's bug loading that I don't know if they 
>> are bugs, or features meant to save memory. This is one of them.
>>
>> The problem I have is that I often need to ask the question "Which 
>> classes implement interface X?" and "which classes are subclasses of Y?"
>> And the current class loading is leaving me high and dry.
>>
>> Here is a simple test case that shows how the current class loading 
>> can
>> break:
>>
>> G.reset();
>> Options.v().set_prepend_classpath(true);
>> Options.v().set_time(false);
>> Scene.v().addBasicClass("java.util.List");
>> Scene.v().loadNecessaryClasses();
>> SootClass arrayList = Scene.v().getSootClass("java.util.ArrayList");
>> Hierarchy h = new Hierarchy();
>>
>> //This test works only if java.util.ArrayList is added as a basic 
>> class //Or if ArrayList is loaded with forceResolve 
>> assertTrue("ArrayList should be an implementer of List", 
>> h.getImplementersOf(list).contains(arrayList));
>>
>> The test is false. There are no classes implementing List as far as 
>> Soot can see.
>>
>> Is this behavior something wanted (a feature)? Or is it a bug?
>>
>> This can be avoided easily by using Scene.forceResolve on all 
>> dandling classes. If this is a bug, I'll open a bug in Github and add 
>> that resolution step. If it is a feature, should we add a 
>> command-line flag for Soot to perform the additional resolution for 
>> people like me who need it?
>>
>> Regards,
>>
>> --
>> Marc-André Laverdière-Papineau
>> Doctorant - PhD Candidate
>> _______________________________________________
>> Soot-list mailing list
>> Soot-list at sable.mcgill.ca
>> http://mailman.cs.mcgill.ca/mailman/listinfo/soot-list
> 
_______________________________________________
Soot-list mailing list
Soot-list at sable.mcgill.ca
http://mailman.cs.mcgill.ca/mailman/listinfo/soot-list