[Soot-list] Optimizing Soot class loading

[Yifei Zhang]

Hi Andreas and Eric,
Thanks for your reply. 
I guess the reason is possibility 2. I am going to try the option 2 next week. Thank you so much for your solution. 
By the way, I place the application jar in the process-dir so that all the classes in that jar are application classes. If every jar file is placed into the process-dir, are they all become application classes?

Cheers,
Yifei

On 2019/1/4, 20:46, "Andreas Peter Dann" <adann at mail.uni-paderborn.de> wrote:

    Hello Yifei,
    
    IMHO: there are two possibilities why the classloading takes more than 70 minutes. Does the loading of the application or library classes take several minutes?
    
    Possibility 1: The zip/jar files are really huge, and thus opening/unzipping takes a lot of time --> IMHO: unlikely.
    
    Possibility 2: Soot does not cache the content of a found jar/zip file.
    Every time a new class is requested, Soot loops over all jar files, opens each jar file, and checks if it contains the class file.
    This may take some time for thousands of dependencies.
    
    To cope with the problem there exist 2 quick fixes:
    
    Option 1: get rid of the dependencies, do not put them on the classpath, and set the option no_bodies_for_excluded 
    
    Option 2: Include the dependencies in the process-dir. In this case, all classes are loaded once at the start.
    Thus, it may take some initial time but Soot only loops once over all jar files.
    Do you get any speed improvements for option 2?
    
    
    Cheers,
    
    Andreas
    
    
    ________________________________________
    Von: Eric Bodden
    Gesendet: Freitag, 4. Januar 2019 12:58
    An: Yifei Zhang
    Cc: soot-list at CS.McGill.CA; Andreas Peter Dann
    Betreff: Re: [Soot-list] Optimizing Soot class loading
    
    Hi Yifei.
    
    My apologies for the late response. This is a _really_ long loading time - quite unusual. I believe that Andreas (in CC) is currently working on an optimized version of the class loading. I wonder if he has any comments on what you are seeing.
    
    libzip.so is external code, which has nothing to do with Soot, though. To me this seems to suggest that the bottleneck may be in your operating system code, or in the way Soot / the Java runtime uses it.
    
    Cheers
    Eric
    
    > On 17. Dec 2018, at 13:33, Yifei Zhang <yifei.zhang1992 at outlook.com> wrote:
    >
    > Dear all,
    >
    > I am using Soot to analyze a Java program with lots of dependencies (thousands of jar files). The application code is in one jar file, and it is specified by the -process-dir option. Other dependencies are specified by the -soot-classpathoption.
    >
    > I found that the class loading is quite slow. It takes about 70 minutes running in a high-performance machine. The time is obtained by measuring the invocation ofScene.v().loadNecessaryClasses() inside the Main.run() method.
    >
    > I looked into the code for class loading of Soot and I guess one possible reason is the worklist algorithm for class loading (in the soot/SootResolver.java) is not optimized. Moreover, after profiling the Soot execution with the perf tool I found that the hot method during class loading is inside the libzip.so. I was wondering if anyone has any idea for optimizing this process. Or could anyone give me some advice or share some experience for optimizing the class loading of Soot?
    >
    > Thanks a lot.
    >
    > Cheers,
    > Yifei
    > _______________________________________________
    > Soot-list mailing list
    > Soot-list at CS.McGill.CA
    > https://mailman.CS.McGill.CA/mailman/listinfo/soot-list