Writing a test script

For each testbench in your test suite, you need to write a test script that MCY can call to run that testbench on a mutated design. MCY will create a temporary directory, place a file named input.txt with a numbered list of mutations in it, run your test script, and expect to find the return status of the testbench for each mutation in a correspondingly numbered list in a file named output.txt. (By default, the list will only contain a single entry, as this is the most straightforward to use. The parameter maxbatchsize can be set in the [test] section of config.mcy to increase the number of mutations included.)

The test script will usually consist of three steps: exporting the mutated source, running the testbench, and reporting the result.

Exporting the Mutated Source

The first step is to obtain the modified source that includes the mutation(s) listed in input.txt. The script create_mutated.sh makes this painless as long as your testbench can accept verilog or rtlil sources for the mutated module. When executing the test, MCY sets the variable $SCRIPTS to the path of the directory where you can find this script.

If you want to substitute a mutated verilog module with identical interface to the original, simply call it with no arguments:

bash $SCRIPTS/create_mutated.sh

This will result in a file named mutated.v containing a mutated module with the same name as the original module being created in the temporary directory where the test is executed (task/<uuid>).

If you want to use the maxbatchsize parameter to test multiple mutations in a single call of the test script, call the script with -c:

bash $SCRIPTS/create_mutated.sh -c

This will result in a file mutated.v with a module of the same name but with an extra input signal mutsel, which you can use to select which mutation to enable.

For more details about mutation generation, see Mutation export options.

Running the Testbench

Substitute the mutated module for the original in your testbench sources (usually by replacing the source file in the sources list with mutated.v). If you are testing a single mutation at a time and did not pass the -c argument to create_mutated.sh, you can now prepare and run your testbench as usual. The mutated module seamlessly replaces the original.

If you did enable multiple mutations to be included in the module, modify your testbench to add the mutsel input to the mutated module. If your testbench is compiled, add a way to pass the value of mutsel to the test at execution, e.g. via command line argument. The first column in input.txt is the number that selects the corresponding mutation. Run the testbench for each value appearing in input.txt.

For example, if using iverilog (from the picorv32_primes example):

iverilog -o sim ../../sim_simple.v mutated.v
while read idx mut; do
        vvp -N sim +mut=${idx} > sim_${idx}.out
done < input.txt

Reporting the Result

The results of the testbench run should be written to output.txt. Each line should be the number identifying the mutation followed by a status. The status can be an arbitrary string not containing whitespace; however, any value not listed with the expect keyword in the test section of config.mcy is considered an error and will cause the mcy run to be aborted. Commonly used return values are “PASS”, “FAIL” and sometimes “TIMEOUT”.

If only a single mutation is evaluated at a time, the associated number is always 1. In this case, simply write 1 followed by the outcome of the testbench to output.txt:

if $test_ok; then
        echo "1 PASS" > output.txt
else
        echo "1 FAIL" > output.txt
fi

If more than one mutation are evaluated in a run, each result should be on its own line, preceded by the corresponding value of mutsel. The order does not matter.

For the previous example with iverilog:

while read idx mut; do
        this_md5sum=$(md5sum sim_${idx}.out | awk '{ print $1; }')
        if [ $good_md5sum = $this_md5sum ]; then
                echo "$idx PASS" >> output.txt
        else
                echo "$idx FAIL" >> output.txt
        fi
done < input.txt