Paper Reading List

【OOPSLA'21】JavaDL: Automatically Incrementalizing Java Bug Pattern Detection (https://creichen.net/papers/javadl.pdf)
【ICSE'22】Detecting False Alarms from Automatic Static Analysis Tools: How Far are We? (https://arxiv.org/abs/2202.05982)
【OOPSLA'19】Coverage Guided, Property Based Testing (https://dl.acm.org/doi/pdf/10.1145/3360607 )
【ASE'20】Legion: Best-first concolic testing (https://dl.acm.org/doi/pdf/10.1145/3293882.3330576 )
【FSE'20】 Metamorphic Testing of Datalog Engines (https://dl.acm.org/doi/pdf/10.1145/3468264.3468573 )
【ISSTA'19】Semantic Fuzzing with Zest (https://dl.acm.org/doi/pdf/10.1145/3293882.3330576)
【ISSTA'21】SAND: A Static Analysis Approach for Detecting SQL Antipatterns (https://dl.acm.org/doi/10.1145/3460319.3464818)
【ISSTA'20】Abstracting Failure-Inducing Inputs (https://havrikov.github.io/publications/issta2020_havrikov.pdf)
【ASPLOS'10】A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs (https://people.cs.rutgers.edu/~sn349/papers/asplos2010.pdf)

说明：
1. 原则上两位同学为一组，选择上面列表中的一篇论文（各小组之间选择的论文不能重复）进行presentation。
2. presentation一般以slides的方式讲解，如果论文有工具，要求使用工具，并做简单演示。
3. presentation的最后需要有几张slides专门讨论小组成员对该研究工作的理解、优点和局限性，以及评价其工具。
4. presentation的质量占课程的主要成绩（考核点：准备是否充分、讲解是否有条理性、对研究工作的见解是否到位）。

Metamorphic Testing(蜕变测试)

Metamorphic testing的两篇综述论文

• Sergio Segura, Gordon Fraser, Ana Belén Sánchez, Antonio Ruiz Cortés: A Survey on Metamorphic Testing. IEEE Trans. Software Eng. 42(9): 805-824 (2016) (https://ieeexplore.ieee.org/document/7422146)
• Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, Zhi Quan Zhou: Metamorphic Testing: A Review of Challenges and Opportunities. ACM Comput. Surv. 51(1): 4:1-4:27 (2018) (https://dl.acm.org/doi/10.1145/3143561, 给出了对MT更深入的分析，解释了一些对MT的误解，强烈推荐！）

Metamorphic testing的简易技术教程

• “Metamorphic Testing 20 Years Later: A Hands‐on Introduction” by Sergio Segura, Zhi Quan Zhou, ICSE 2018, Technical Briefing （https://zenodo.org/record/1256230/files/ICSE18-TB-Segura-Zhou.pdf， 对MT的相关概念进行了梳理，有些内容来源于上面的两篇综述论文）

如何找到Metamorphic Relation (MR)？（这是应用Metamorphic Testing的关键问题）

• Andrei Lascu, Matt Windsor, Alastair F. Donaldson, Tobias Grosser, John Wickerson: Dreaming up Metamorphic Relations: Experiences from Three Fuzzer Tools. MET@ICSE 2021: 61-68 （给出了一些如何找到MR的经验性原则）

MT的典型应用论文：

[编译器测试]
• Vu Le, Mehrdad Afshari, Zhendong Su: Compiler validation via equivalence modulo inputs. PLDI 2014: 216-226 (Complier Testing)
[APP测试]
• Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, Zhendong Su: Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs. Proc. ACM Program. Lang. 5(OOPSLA): 1-31 (2021)
• Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, Zhendong Su: Understanding and finding system setting-related defects in Android apps. ISSTA 2021: 204-215
[数据库测试]
• Manuel Rigger, Zhendong Su: Testing Database Engines via Pivoted Query Synthesis. OSDI 2020: 667-682
• Manuel Rigger, Zhendong Su: Finding bugs in database systems via query partitioning. Proc. ACM Program. Lang. 4(OOPSLA): 211:1-211:30 (2020)
• Manuel Rigger, Zhendong Su: Detecting optimization bugs in database engines via non-optimizing reference engine construction. ESEC/SIGSOFT FSE 2020: 1140-1152
[SMT测试]
• Dominik Winterer, Chengyu Zhang, Zhendong Su: Validating SMT solvers via semantic fusion. PLDI 2020: 718-730
[其他领域]
• Andrei Lascu, Alastair F. Donaldson, Tobias Grosser, Torsten Hoefler: Metamorphic Fuzzing of C++ Libraries. ICST 2022: 35-46

Property-based Testing(基于性质的测试)

Property-based Testing的经典论文：

• Fink, George, and Matt Bishop. “Property-based testing: a new approach to testing for assurance.” ACM SIGSOFT Software Engineering Notes 22.4 (1997): 74-80. https://dl.acm.org/doi/pdf/10.1145/263244.263267
• Koen Claessen, John Hughes: QuickCheck: a lightweight tool for random testing of Haskell programs. ICFP 2000: 268-279 https://dl.acm.org/doi/10.1145/357766.351266

Property-based Testing的应用论文

[Mobile Apps测试]
• Lam, Edmund SL, Peilun Zhang, and Bor-Yuh Evan Chang. “ChimpCheck: property-based randomized test generation for interactive apps.” Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 2017. https://github.com/cuplv/chimpcheck
[Web Apps测试]
• O‘Connor, Liam, and Oskar Wickström. “Quickstrom: property-based acceptance testing with LTL specifications.” Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 2022. https://github.com/quickstrom/quickstrom
[ROS系统测试]
• André Santos, Alcino Cunha, Nuno Macedo: Property-based testing for the robot operating system. A-TEST@ESEC/SIGSOFT FSE 2018: 56-62 https://dl.acm.org/doi/10.1145/3278186.3278195
[Java SDK Library测试]
• Rohan Padhye, Caroline Lemieux, Koushik Sen: JQF: coverage-guided property-based testing in Java. ISSTA 2019: 398-401 https://dl.acm.org/doi/pdf/10.1145/3293882.3339002 （工具： https://github.com/rohanpadhye/jqf ）
[其他一些软件系统的测试]
• Andreas Stahlbauer, Marvin Kreis, Gordon Fraser: Testing scratch programs automatically. ESEC/SIGSOFT FSE 2019: 165-175 https://dl.acm.org/doi/10.1145/3338906.3338910
• Stefan Karlsson, Adnan Causevic, Daniel Sundmark: QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs. ICST 2020: 131-141 https://ieeexplore.ieee.org/document/9159071/
• Andreas Löscher, Konstantinos Sagonas: Targeted property-based testing. ISSTA 2017: 46-56 https://dl.acm.org/doi/10.1145/3092703.3092711

可以理解为也属于Property-based Testing的应用论文

• Park M, Jang H, Byun T, et al. Property-based testing for LG home appliances using accelerated software-in-the-loop simulation[C]//2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2020: 120-129.
• Ruijie Meng, Zhen Dong, Jialin Li, Ivan Beschastnikh, and Abhik Roychoudhury. 2022. Linear-time temporal logic guided greybox fuzzing. In Proceedings of the 44th International Conference on Software Engineering (ICSE '22). Association for Computing Machinery, New York, NY, USA, 1343–1355. https://doi.org/10.1145/3510003.3510082

Property-based Testing的talks：

• Oskar Wickström - Property-Based Testing The Ugly Parts: Case Studies from Komposition https://www.youtube.com/watch?v=z2ete8VZnZY
• Hughes - Don‘t Write Tests. https://www.youtube.com/watch?v=hXnS_Xjwk2Y
• John Hughes - Testing the Hard Stuff and Staying Sane. https://www.youtube.com/watch?v=zi0rHwfiX1Q

Property-based Testing的一些其他参考资料（都讲的很好）

• Overview of Property-Based Testing Functionality，https://github.com/jmid/pbt-frameworks (很不错的一个资源，是一个丹麦的教授维护的，他自己也教授functional programming, 汇集了不少PBT的框架比较和资源)
• Model-based Testing (Stateful and Model-based Properties with Examples in Java): https://johanneslink.net/model-based-testing/  （这篇技术博客把广义上的model-based testing讲的非常清楚）
• Relational Problems to Teach Property-Based Testing (by John Wrenna , Tim Nelsona , and Shriram Krishnamurthia) https://cs.brown.edu/~tbn/publications/wnk-pj20-pbt.pdf
• Property-based Testing and Test Oracles https://swen90006.github.io/notes/Property-based-testing.html#    （澳洲一个大学的测试课教程，Section 6.4.3 - Metamorphic oracles, Section 6.4.7 - Oracle derivation, Section 6.5 - Property-based testing。关于MT和PBT这块讲解的挺清楚，其中谈到Metamorphic Testing（MT)是属于Property-based testing（PBT)的一种测试方法）

与Property-based Testing工具相关的资料：

• MacIver D R, Hatfield-Dodds Z. Hypothesis: A new approach to property-based testing[J]. Journal of Open Source Software, 2019, 4(43): 1891. https://joss.theoj.org/papers/10.21105/joss.01891.pdf
• junit-quickcheck: https://github.com/pholser/junit-quickcheck
• Hedgehog: https://hedgehog.qa/
• hypothesis: https://github.com/HypothesisWorks/hypothesis/