[Soot-list] Soot cannot get the body and the callers of the overloaded method

Wed Mar 1 04:22:22 EST 2017

Hi Yury,

Soot supports reading in dex files and converting them to Jimple. Soot does, however, not support modelling the Android lifecycle which is a prerequisite for creating a callgraph unless you are satisfied with very coarse over-approximations such as treating every public method as a potential entry point. I actually thought you were analyzing an Android app together with the framework, i.e., use some app as the driver for your framework analysis.

If you have main() methods, you should specify them as entry points using Scene.v().setEntryPoints() before you start the callgraph analysis with PackManager.v().runPacks(). Note that there might be some limits to the completeness of the callgraph for those parts in which Android uses reflection or passes data through native methods. I actually haven’t tried a full-fledged framework analysis on Android yet. Let me know how well that works. Especially those cases in which Android evaluates configuration files to identify the methods to then call via reflection might be an issue.

You only need FlowDroid if you need to construct an artificial dummy main() method. If you are fine with those main() methods you already find in the Android code, you don’t need it.

Best regards,
  Steven

From: Dr. Yury Zhauniarovich [mailto:yzhauniarovich at hbku.edu.qa]
Sent: Wednesday, March 1, 2017 7:53 AM
To: Arzt, Steven <steven.arzt at sit.fraunhofer.de>
Cc: soot-list at cs.mcgill.ca
Subject: Re: [Soot-list] Soot cannot get the body and the callers of the overloaded method

Good morning Steven,

Thanks for your reply. However, I do not analyze Android applications, I analyze Android framework. As far as I understand, I do not need Flowdroid for this  because Soot already supports dex files. In the case of system services, there are entry points, e.g., com.android.server.SystemServer contains main method, as well as a set of other services. Do you think that I still need Flowdroid to perform the analysis?

Thank you for the remind about your thesis. I've seen it among my GScholar recommendations and wanted to read it, but I've forgotten to download it.
--
Best Regards,
Yury Zhauniarovich

On Tue, 2017-02-28 at 11:55 +0000, Arzt, Steven wrote:
Hi Yury,

You can read up on the concept of library classes vs. application classes here: https://github.com/Sable/soot/wiki/Introduction:-Soot-as-a-command-line-tool. The whole article is a bit dated and focuses on Java, not Android, but the concepts are still pretty much the same aside from the usual surprise here and there.

For the callgraph, you have hit a more substantial issue. Soot’s callgraph algorithms start from an entry point. For a Java program, that is the main() method. Android apps, on the other hand, do not have a main() method. Consequently, there is nothing the callgraph builder could work on (aside from some special cases such as JVM handlers, etc.) To solve this problem, you can use FlowDroid to construct a dummy main() method for you. This dummy main method() is equivalent to a pseudo-implementation of the Android OS for the special case of callgraph construction. It is not executable, it is just good enough to tell the callgraph builder when which method from the app’s individual components are called and in which context. If you want to read up on the details, it’s in my PhD thesis: http://tuprints.ulb.tu-darmstadt.de/5937/7/Thesis.pdf, section 4.15 and pretty much all of section 5. If you just want to get something that works, download soot-infoflow and soot-infoflow-android from Github and try this:

       public static void main(String[] args) throws IOException, XmlPullParserException {
              // Initialize Soot
              SetupApplication analyzer = new SetupApplication("C:\\Program Files (x86)\\Android\\android-sdk\\platforms<file://Android/android-sdk/platforms>",
                           "D:/org.fdroid.k9_17046.apk");
              analyzer.getConfig().setTaintAnalysisEnabled(false);
              analyzer.calculateSourcesSinksEntrypoints(Collections.emptySet(), Collections.emptySet());
              analyzer.runInfoflow();

              // Iterate over the callgraph
              for (Iterator<Edge> edgeIt = Scene.v().getCallGraph().iterator(); edgeIt.hasNext(); ) {
                     Edge edge = edgeIt.next();

                     SootMethod smSrc = edge.src();
                     Unit uSrc = edge.srcStmt();
                     SootMethod smDest = edge.tgt();

                     System.out.println("Edge from " + uSrc + " in " + smSrc + " to " + smDest);
              }
       }

That should give you a fairly complete callgraph of the app. There are various knobs you can turn here and there to make trade-offs between precision, completeness, and performance if you need to. The full story is in my PhD thesis. The most important option is probably whether you want the callbacks to be included or not. Analyzing the callbacks is the process that takes most of the time. The configuration object (returned by getConfig()) has an option setEnableCallbacks() for that.

Best regards,
  Steven

From: Dr. Yury Zhauniarovich [mailto:yzhauniarovich at hbku.edu.qa]
Sent: Tuesday, February 28, 2017 12:27 PM
To: Arzt, Steven <steven.arzt at sit.fraunhofer.de<mailto:steven.arzt at sit.fraunhofer.de>>; loscoccia at gmail.com<mailto:loscoccia at gmail.com>
Cc: soot-list at cs.mcgill.ca<mailto:soot-list at cs.mcgill.ca>
Subject: Re: [Soot-list] Soot cannot get the body and the callers of the overloaded method

Steven,

You were right. retrieveActiveBody() does the trick even without addBasicClass() method. However, the set of calling methods is still empty for the second case. Can you please point me what optimizations should I disable in order to list all methods that call second addService method (now I obtain an empty set)?

One more question, is there any source where I can read about these optimizations, besides the sources of Soot:)?

Thank you in advance!

--
Best Regards,
Yury Zhauniarovich

On Tue, 2017-02-28 at 10:37 +0000, Arzt, Steven wrote:
Hu Yury,

Did you add the addBasicClass() call before calling loadNecessaryClasses()? Did you try retrieveActiveBody() instead of getActiveBody()?

The problem is most likely not with overloads, but with on-demand loading. As I wrote, Soot only loads what is ultimately necessary. That means, if the callgraph algorithm, for example, accesses some overload A of the method, but not some other overload B, it will only load A’s body, and not the one of B, even though both are overloads of the same method. What you need to do is tell Soot that you want the respective body anyway.

Best regards,
  Steven

From: Dr. Yury Zhauniarovich [mailto:yzhauniarovich at hbku.edu.qa]
Sent: Tuesday, February 28, 2017 11:29 AM
To: Arzt, Steven <steven.arzt at sit.fraunhofer.de<mailto:steven.arzt at sit.fraunhofer.de>>; loscoccia at gmail.com<mailto:loscoccia at gmail.com>
Cc: soot-list at cs.mcgill.ca<mailto:soot-list at cs.mcgill.ca>
Subject: Re: [Soot-list] Soot cannot get the body and the callers of the overloaded method

Hi Gian Luca, Steven,

First of all, thank you for your tips. Gian Luca,
Where did you get the android framework files? Keep in mind that those that those that come with Flowdroid (https://github.com/secure-software-engineering/soot-infoflow-android) only contain stubs/summaries of many methods. You have to get your own android jar files.

To get the corresponding jar and dex filex I've build Android, so I know that they contain all the information inside.

Steven, I've added the following code into my app:
SootClass clz = Scene.v().getSootClass("android.os.ServiceManager");
System.out.println("IsPhantom: " + clz.isPhantom());
System.out.println("Resolving Level: " + clz.resolvingLevel());

The output is the following:
IsPhantom: false
Resolving Level: 3

It seems that the output is the one that you've assumed. However, the error is still there. Moreover, this does not explains why I get the correct data for the first method, but not for the second overloaded method. I guess there maybe a bug in Soot dealing with the overloaded methods.

--
Best Regards,
Yury Zhauniarovich

On Tue, 2017-02-28 at 09:54 +0000, Arzt, Steven wrote:
Hi Yuri,

There are several possible explanations for the behavior you encounter. One is that Soot didn’t actually find the class. Can you check whether the class ServiceManager is a phantom, i.e., isPhantom() returns true on the SootClass? If that is the case, the class is probably missing from the JAR file you are using.

An alternative explanation would be that the class is there, but is not loaded. The easiest solution would be to do something like this right before you call loadNecessaryClasses():

                Scene.v().addBasicClass(“android.os.ServiceManager”, SootClass.BODIES);

That line tells Soot to load this class and all method bodies inside it, even if would normally be excluded for some reason. To see whether a class was loaded (and to which extent), check the “resolvingLevel” property of the SootClass. You need level 3 (=SootClass.BODIES) to be able to have bodies. Since method bodies are loaded on demand, you might have level 3 and no bodies in some cases. In that case, SootMethod.retrieveActiveBody() should do the trick. That might all sound complex at first, but the core idea is that Soot only loads what is ultimately necessary to improve speed and reduce memory pressure.

Best regards,
  Steven

From: Soot-list [mailto:soot-list-bounces at cs.mcgill.ca] On Behalf Of Gian Luca Scoccia
Sent: Tuesday, February 28, 2017 10:44 AM
To: Dr. Yury Zhauniarovich <yzhauniarovich at hbku.edu.qa<mailto:yzhauniarovich at hbku.edu.qa>>
Cc: soot-list at CS.McGill.CA<mailto:soot-list at CS.McGill.CA>
Subject: Re: [Soot-list] Soot cannot get the body and the callers of the overloaded method

Where did you get the android framework files? Keep in mind that those that those that come with Flowdroid (https://github.com/secure-software-engineering/soot-infoflow-android) only contain stubs/summaries of many methods. You have to get your own android jar files.

2017-02-28 8:26 GMT+01:00 Dr. Yury Zhauniarovich <yzhauniarovich at hbku.edu.qa<mailto:yzhauniarovich at hbku.edu.qa>>:
Dear community,

I faced with a problem and do not know how to solve an issue and what is its cause. I would be glad if someone can point me for a direction.

I have the following issue with Soot. I try to analyze the Android framework files. In particular, I would like to find all places in the code where system services are added, the corresponding method signatures are defined in the smAddMethodSignatures array (see the code in the end).

When I ran the following code, I am able to find the callers and the body of the method for the first signature but not for the second. For the second signature, the active body is an empty string, and there are no caller, although from the sources it is obvious that such methods exist. Here is the error when I try to get active body:

Exception in thread "main" java.lang.RuntimeException: no active body present for method <android.os.ServiceManager: void addService(java.lang.String,android.os.IBinder,boolean)>

       at soot.SootMethod.getActiveBody(SootMethod.java:323)

       at com.tmp.BAnalysisApp.getRegisteredServicesClasses(BAnalysisApp.java:85)

       at com.tmp.BAnalysisApp.main(BAnalysisApp.java:47)

I am a novice with Soot, and I may miss something. However, it seems to me that there is a bug in Soot analyzing overloaded methods. I also filled an issue in the tracker with a pretty 700 number: https://github.com/Sable/soot/issues/700

Here is the code of an MWE that I use:

import java.util.Collections;

import java.util.Iterator;

import java.util.List;

import soot.PackManager;

import soot.Scene;

import soot.SootClass;

import soot.SootMethod;

import soot.jimple.toolkits.callgraph.CallGraph;

import soot.jimple.toolkits.callgraph.Edge;

import soot.options.Options;

public class BAnalysisApp {

    private final static String SERVICES_DEX_DIR_PATH = "/home/yury/tmp/tmp_services/";

    private final static String ANDROID_BOOT_JAR_PATH = "/home/yury/tmp/android-boot-25.jar";

    // searched method signatures

    // public static void addService(String name, IBinder service)

    // public static void addService(String name, IBinder service, boolean allowIsolated)

    private final static String[] smAddMethodSignatures = {

           "<android.os.ServiceManager: void addService(java.lang.String,android.os.IBinder)>",

           "<android.os.ServiceManager: void addService(java.lang.String,android.os.IBinder,boolean)>" };

    public static void main(String[] args) {

       prepareSoot();

       List<SootClass> registeredServices = getRegisteredServicesClasses();

    }

    private static void prepareSoot() {

       soot.G.reset();

       Options.v().set_src_prec(Options.src_prec_apk);

       Options.v().set_process_dir(Collections.singletonList(SERVICES_DEX_DIR_PATH));

       Options.v().set_process_multiple_dex(true);

       Options.v().set_force_android_jar(ANDROID_BOOT_JAR_PATH);

       Options.v().set_whole_program(true);

       Options.v().set_allow_phantom_refs(true);

       Options.v().set_output_format(Options.output_format_none);

       Options.v().setPhaseOption("cg.spark", "on");

       Scene.v().loadNecessaryClasses();

       PackManager.v().runPacks();

    }

    private static List<SootClass> getRegisteredServicesClasses() {

       final CallGraph cg = Scene.v().getCallGraph();

       for (String mthSig : smAddMethodSignatures) {

           SootMethod smAddServiceMth = Scene.v().grabMethod(mthSig);

           System.out.println(mthSig);

            //printing the body

           System.out.println(smAddServiceMth.getActiveBody().toString());

            //iterating over the caller methods

           Iterator<Edge> edgeIterator = cg.edgesInto(smAddServiceMth);

           while (edgeIterator.hasNext()) {

               Edge mtdEdge = edgeIterator.next();

               SootMethod srcMtd = mtdEdge.src();

               System.out.println(srcMtd.getSignature());

               System.out.println(mtdEdge.srcStmt().toString());

           }

       }

       return null;

    }

}

--
Best Regards,
Yury Zhauniarovich

CONFIDENTIALITY NOTICE:
This email and any attachments transmitted with it are confidential and intended for the use of individual or entity to which it is addressed. If you have received this email in error, please delete it immediately and inform the sender. Unless you are the intended recipient, you may not use, disclose, copy or distribute this email or any attachments included. The contents of this email, including any attachments, may be subjected to copyright law. In such cases, the contents may not be copied, adapted, distributed or transmitted without the consent of the copyright owner.

_______________________________________________
Soot-list mailing list
Soot-list at CS.McGill.CA<mailto:Soot-list at CS.McGill.CA>
https://mailman.CS.McGill.CA/mailman/listinfo/soot-list

--
_______________________________________________________

Ph.D. Student at Gran Sasso Science Institute (GSSI)<http://www.gssi.infn.it>
Personal Page<http://cs.gssi.infn.it/people/scoccia/>
Linkedin Profile<https://it.linkedin.com/in/gianlucascoccia>

CONFIDENTIALITY NOTICE:
This email and any attachments transmitted with it are confidential and intended for the use of individual or entity to which it is addressed. If you have received this email in error, please delete it immediately and inform the sender. Unless you are the intended recipient, you may not use, disclose, copy or distribute this email or any attachments included. The contents of this email, including any attachments, may be subjected to copyright law. In such cases, the contents may not be copied, adapted, distributed or transmitted without the consent of the copyright owner.

CONFIDENTIALITY NOTICE:
This email and any attachments transmitted with it are confidential and intended for the use of individual or entity to which it is addressed. If you have received this email in error, please delete it immediately and inform the sender. Unless you are the intended recipient, you may not use, disclose, copy or distribute this email or any attachments included. The contents of this email, including any attachments, may be subjected to copyright law. In such cases, the contents may not be copied, adapted, distributed or transmitted without the consent of the copyright owner.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.CS.McGill.CA/pipermail/soot-list/attachments/20170301/57aeb6d6/attachment-0001.html>