Research on Mobile App Analysis and Testing

Background

Mobile apps are typical GUI-centered and event-driven software. They are now ubiquitous and serving the needs of our daily life in many different aspects. However, due to the complex end-user environments (e.g., different OSes, vendor devices and third-party libraries), ensuring app reliability and correctness has thus become a longstanding challenge in both academia and industry (see literature). Our research aims to tackle this challenge by developing novel, effective and practical approaches and techniques to improve app quality, reliability and usability.

Techniques, Tools and Dataset

To this end, we have devoted much research effort over the recent years and developed several effective app analysis and testing techniques, including:
  • Stoat, a fully automated GUI fuzzing technique for finding crashing bugs;
  • Genie, SetDroid and Odin, fully automated GUI fuzzing techniques for finding non-crashing, logic bugs;
  • SetChecker, a static analysis tool for finding system setting related bugs;
  • Themis, the first ground-truth benchmark for evaluating/analyzing automated GUI fuzzing tools;
  • DroidDefects/CrashAnalysis: the dataset of framework-specific exception bugs of Android apps.

    • Research Impact

    In addition to successfully finding many bugs in open-source apps, our techniques have found and reported 100+ bugs in several highly-popular industrial apps with billions of monthly-active users, many of which have been already fixed by the app vendors. For example:

    TikTok (Douyin): 78 (confirmed) / 32 (fixed)
    WeChat: 11 (confirmed) / 11 (fixed)
    CapCut: 4 (confirmed) / 4 (fixed)
    QQmail: 2 (confirmed) / 2 (fixed)
    Google+: 2 (confirmed) / 2 (fixed)
    AlipayHK: 2 (confirmed) / 2 (fixed)
    Gmail: 1 (confirmed) / 1 (fixed)

    Up to now:
    - Stoat has become a representative model-based testing approach for Android (cited by 225+), and used/compared/extended by many work);
    - SetDroid has been intergated into FastBot (ByteDance's official app testing infrastructure) for daily testing (see this post from ByteDance's SE Lab);
    - Themis has helped optimize/enhance FastBot with several new GUI fuzzing & mutation strategies.

    Selected Publications

  • Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning
    Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, Ping Yang
    37th IEEE/ACM International Conference on Automated Software Engineering
    ASE 2022 (industry track), pdf.

  • Detecting Non-crashing Functional Bugs in Android Apps via Deep-State Differential Analysis
    Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, Zhendong Su
    ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
    ESEC/FSE 2022, pdf.

  • Fully Automated Functional Fuzzing of Android Apps for Detecting Non-Crashing Logic Bugs
    Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, Zhendong Su
    ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications
    SPLASH/OOPSLA 2021, pdf, talk video, Genie.

  • Benchmarking Automated GUI Testing for Android against Real-World Bugs
    Ting Su, Jue Wang, Zhendong Su
    29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
    ESEC/FSE 2021, pdf, talk video, Themis.

  • Understanding and Finding System Setting-Related Defects in Android Apps
    Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, Zhendong Su
    30th ACM SIGSOFT International Symposium on Software Testing and Analysis
    ISSTA 2021, pdf, SetDroid.

  • Why My App Crashes? Understanding and Benchmarking Framework-specific Exceptions of Android apps
    Ting Su, Lingling Fan, Sen Chen, Yang Liu, Lihua Xu, Geguang Pu, Zhendong Su
    IEEE Transactions on Software Engineering
    TSE 2020, pdf, DroidDefects.

  • Efficiently Manifesting Asynchronous Programming Errors in Android Apps
    Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu
    The 33rd IEEE/ACM International Conference on Automated Software Engineering
    ASE 2018, pdf.

  • Large-Scale Analysis of Framework-Specific Exceptions in Android Apps
    Lingling Fan#, Ting Su#, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu and Zhendong Su
    The 40th International Conference on Software Engineering
    ICSE 2018, pdf, slides, CrashAnalysis.
    (#Equal Contribution)

    •     ACM SIGSOFT Distinguished Paper Award

  • Guided, Stochastic Model-Based GUI Testing of Android Apps
    Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, Zhendong Su
    The 11th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering
    ESEC/FSE 2017, pdf, slides, Stoat.

    •    Best Research Prototype Tool Award (NASAC 2017 held by CCF)

    Other Publications

  • SetDroid: Detecting User-configurable Setting Issues of Android Apps via Metamorphic Fuzzing
    Jingling Sun
    The 43th International Conference on Software Engineering
    ICSE 2021, ACM Student Research Competition, pdf

       Second Place of ACM Student Research Competition

  • FSMdroid: Guided GUI Testing of Android Apps
    Ting Su
    The 38th International Conference on Software Engineering
    ICSE 2016, ACM Student Research Competition, pdf, Press

       Golden Medal (First Place) of ACM Student Research Competition

    • Under Review

  • Characterizing and Finding System Setting-Related Defects in Android Apps
    Jingling Sun, Ting Su, Kai Liu, Chao Peng, Zhao Zhang, Geguang Pu, Tao Xie, Zhendong Su
    IEEE Transactions on Software Engineering (under review)



    • Acknowledge: Our research has received generous funding support from Chinese NSF, Swiss NSF, Google, ByteDance, and NTUitive Gap Fund.

    If you have some questions or want to know more, feel free to contact us (Ting Su).

    last modified: 2022.8.8