MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation

Highlights

• Automatically deriving metamorphic relations is relevant for software testing.

• MeMo is an approach to automatically identify MRs in natural language code comments.

• The MRs get automatically translated into executable assertions.

• MeMo’s oracles improve automated testing when used for bug detection.

• They discover bugs not only at the interface level, but also at deeper levels.

Abstract

Software testing depends on effective oracles. Implicit oracles, such as checks for program crashes, are widely applicable but narrow in scope. Oracles based on formal specifications can reveal application-specific failures, but specifications are expensive to obtain and maintain. Metamorphic oracles are somewhere in-between. They test equivalence among different procedures to detect semantic failures. Until now, the identification of metamorphic relations has been a manual and expensive process, except for few specific domains where automation is possible. We present MeMo, a technique and a tool to automatically derive metamorphic equivalence relations from natural language documentation, and we use such metamorphic relations as oracles in automatically generated test cases. Our experimental evaluation demonstrates that 1) MeMo can effectively and precisely infer equivalence metamorphic relations, 2) MeMo complements existing state-of-the-art techniques that are based on dynamic program analysis, and 3) metamorphic relations discovered with MeMo effectively detect defects when used as test oracles in automatically-generated or manually-written test cases.