Writing Properties - A new approach to testing

Writing smart contract properties - A new approach to testing

Scribble allows you to write smart contract properties that can be automatically tested using fuzzing and symbolic execution techniques. Writing properties requires a bit of a mindset shift. This article will talk about that shift, to go from unit testing to property-based testing with Scribble.

If you’re a developer then I’m sure you’re familiar with unit testing, an approach where you write small (unit) test cases to see if a component behaves as expected.

While it is possible (and easy) to deviate, tests will usually follow the following general structure:

Arrange - A unit test will first set up the program in a state where we can test the desired feature.

Act - The unit test will then act on the program

Assert - Finally the unit test will check if the program behaved as expected

// Arrange
token = new ERC20("ExampleToken", 20)
recipient = accounts[1]

// Act
token.transfer(recipient, 10)

// Assert
assert.equal(token.balanceOf(recipient), 10, "the recipient has 10 ExampleTokens")

The problem with unit testing

When unit testing, you’ll have several test cases for the same function/ component, testing various edge cases.

Take the following function which adds two numbers:

function add (a: uint, b: uint) returns (uint) {
	// We'll add a very obvious bug here that should be caught:
	if (a == b) {
		return a + b + 1;
	}
	return a + b;
}

If we were to write unit tests for this function, then we’d come up with several input scenarios:

We’ll try different cases like:

1. Add two positive numbers
2. What if we use 0 as a value for one of the inputs
3. What if the numbers are equal

In this case, we would catch the bug with our third test case, where we try two equal numbers and see that the input is 201.

Unfortunately, this approach is not ideal:

1. It takes a lot of work to come up with the different test cases needed to cover the contract appropriately, even though we’re testing relatively simple code here.
2. This is a simple example. In the real world, it will be much more challenging to cover the right behaviours.

Want to make sure your tests cover the right edge cases? Use mutation testing for better insight into your test quality!

Properties

Let’s look at writing properties instead of coming up with lots of test cases. We’ll describe the behaviour we want to test with specifications.

I’ll use the example from above. All the test cases, test the same thing (a single property 😃). Namely: “The result is the sum of the inputs”. In fact, we came up with several concrete cases and their expected results.

Writing properties means we don’t take this last step; instead, we formalise the expected behaviour using an annotation above the function:

/// if_succeeds {:msg "The result is the sum of the inputs"} $result == a + b;
function add (a: uint, b: uint) returns (uint) {
	// We'll add a very obvious bug here that should be caught:
	if (a == b) {
		return a + b + 1;
	}
	return a + b;
}

📚 Find out more about function specifications here!

You can now use Scribble to instrument the contract and add code to check this property and raise an assertion whenever the property is violated.

Instead of needing to come up with various test cases with inputs and expected outputs, we’ll only need to come up with the inputs. We could even automatically run the code with millions of different inputs by using fuzzing.

Scribble and Testing

Hear me out before you go and delete your unit tests! They are still invaluable.

1. Unit tests provide a baseline Automatic tools can struggle to reach all the parts of your codebase. Having unit tests will ensure a strong baseline helping you catch bugs and regressions before they end up on chain!

2. Unit tests are good for triaging - Unit tests are super helpful for triaging bugs and validating a fix. Fuzzing will take a generalised approach to cover your codebase, while unit tests will test different behaviours in isolation!

3. Unit tests are fast - When you’re coding you want instant feedback seeing that you haven’t broken any previous tests. Rigorous fuzz testing can take hours or days, not something you have time for if you need to get this feature in by Friday.

4. Unit Tests and Scribble are complementary - You can use Scribble properties to enhance existing unit test; scribble instruments the smart contracts so that assertions are raised whenever a property is violated. Unit testing can also check these assertions! Your existing test suite covers many different behaviours and tests them with assertions. You can combine that with Scribble to check both the manually written assertions and the properties written in Scribble.

In short, fuzzing is a killer extension to your current testing & security tools & techniques.

Fin

Maybe you’ll end up experimenting with Scribble after reading this article. If so, then let us know what you think about the experience! Scribble is still very new (only released one month ago!) and we’re very eager to receive feedback so we can improve Scribble!

Do you have any questions? Come by on Discord, and we’ll try to help you!

Feel free to open up a GitHub issue for feature suggestions, bug reports or any other code related comments!

Thanks to Carl Fartherson for feedback on a draft of this article!

All posts