Hannah Potter

Mutation testing is an effective testing technique for improving how well a test suite can detect small changes to a program under test. This testing technique is seeing increased industry adoption. This paper aims to study the use of mutation testing in an educational setting and understand students’ technical and conceptual challenges in applying mutation testing concepts. We report on two case studies of incorporating mutation testing into software engineering curricula.The Scaffolding Study explores the impact of using different mutation analysis tools directly or indirectly via a uniform interface provided by an educational infrastructure. We observe that scaffolding (indirect tool use) improved the consistency of student performance for those using the same mutation analysis tool on the same code as well as helping students perform more effective mutation testing.The Visualization Study explores the impact of different forms of output of a mutation analysis tool. Specifically, it assesses to what extent visualizations support students in reasoning about mutants and writing tests to detect them. We observe that like scaffolding, visualizations helped students perform more effective mutation testing, with lower-performing students seeing a boost in particular.We further explore challenges around automatic assessment of mutation testing exercises. For example, we observe that even with assignment scaffolding, 18–21% of student submissions required manual modifications to successfully execute.

Learning the syntax and semantics of a new programming language is a challenge. It is common for learners to refer to language documentation many times and in many contexts as they build comfort and understanding. We review existing functional language documentation, finding that it tends to be organized according to the structure of the language. Each section interleaves narrative explanations, which introduce precise terminology that is then used consistently, with code examples. Sections often start with simpler special cases of a construct before considering it in full generality. To make use of language documentation, learners must step away from the code they are working with, e.g., in an exercise or tutorial, to locate and transfer knowledge from the documentation. We describe a system, ExplainThis, that automatically generates contextualized language documentation, structured based on our study of language documentation but specialized to the particular code at the cursor. This system is integrated into the structure editor of Hazel, a live functional environment. Documentation appears next to the editor and color is used as secondary notation to correlate the explanation with program terms. We also study syntactic and explanatory specificity with a formative user study. We find that participants desire documentation to be tailored to specific syntax of the code a user is working with, while allowing an adaptive level of specificity for code examples.