Merge pull request #188 from crytic/dev-exercise-multi-abi

Add tutorial for multi-abi

Author: montyly

.github/workflows/echidna.yml

@@ -117,6 +117,14 @@ jobs:
             contract: TestDepositWithPermit
             outcome: success
             expected: 'testERC20PermitDeposit(uint256):\s*passed'
+          - name: MultiABI
+            workdir: program-analysis/echidna/example/
+            files: multiabi.sol
+            solc-version: 0.8.0
+            config: multiabi.yaml
+            contract: EchidnaTest
+            outcome: failure
+            expected: 'test_flag_is_false():\s*failed'
 
     steps:
     - name: Checkout repository

program-analysis/echidna/README.md

@@ -25,6 +25,7 @@ Watch our [Fuzzing workshop](https://www.youtube.com/watch?v=QofNQxW_K08&list=PL
   - [How to test bytecode-only contracts](./testing-bytecode.md): How to fuzz a contract without bytecode or to perform differential fuzzing between Solidity and Vyper
   - [How to use hevm cheats to test permit](./hevm-cheats-to-test-permit.md): How to test code that depends on ecrecover signatures using hevm cheat codes
   - [How to seed Echidna with unit tests](./end-to-end-testing.md): How to use existing unit tests to seed Echidna
+  - [Understanding and using `multi-abi`](./using-multi-abi.md): What is `multi-abi` testing, and how can it be used
   - [Fuzzing tips](./fuzzing_tips.md): General fuzzing tips
 - Exercises
   - [Exercise 1](./Exercise-1.md): Testing token balances

program-analysis/echidna/example/multiabi.sol

@@ -0,0 +1,32 @@
+pragma solidity ^0.8.0;
+
+contract Flag {
+
+   bool flag = false;
+
+   function flip() public {
+       flag = !flag;
+   }
+
+   function get() public returns (bool) {
+        return flag;
+   }
+
+   function test_fail() public {
+       assert(false);
+   }
+}
+
+
+contract EchidnaTest {
+   Flag f;
+
+   constructor() {
+      f = new Flag();
+   }
+
+   function test_flag_is_false() public {
+      assert(f.get() == false);
+   }
+
+}

program-analysis/echidna/example/multiabi.yaml

@@ -0,0 +1,3 @@
+testMode: assertion
+testLimit: 50000
+multi-abi: true

program-analysis/echidna/using-multi-abi.md

@@ -0,0 +1,122 @@
+# Understanding and using `multi-abi` in Echidna
+
+**Table of contents:**
+
+- [Introduction](#introduction)
+- [What is `multi-abi` testing?](#what-is-multi-abi-testing)
+- [When and how to use `multi-abi`](#when-and-how-to-use-multi-abi)
+- [Run Echidna](#run-echidna)
+- [Use cases and conclusions](#use-cases-and-conclusions)
+
+## Introduction
+
+This tutorial is written as a hands-on guide to using `multi-abi` testing in Echidna. You will learn what `multi-abi` testing is, how to use it in your tests, and what to expect from its usage.
+
+## What is `multi-abi` testing?
+
+It is a testing mode that allows Echidna to call functions from any contract not directly under test. The ABI for the contract must be known, and it must have been deployed by the contract under test.
+
+## When and how to use `multi-abi`
+
+By default, Echidna calls functions from the contract to be analyzed, sending the transactions randomly from addresses `0x10000`, `0x20000` and `0x30000`.
+
+In some systems, the user has to interact with other contracts prior to calling a function on the fuzzed contract. A common example is when you want to provide liquidity to a DeFi protocol, you will first need to approve the protocol for spending your tokens. This transaction has to be initiated from your account before actually interacting with the protocol contract.
+
+A fuzzing campaign meant to test this example protocol contract won't be able to modify users allowances, therefore most of the interactions with the protocol won't be tested correctly.
+
+This is where `multi-abi` testing is useful: It allows Echidna to call functions from other contracts (not just from the contract under test), sending the transactions from the same accounts that will interact with the target contract.
+
+## Run Echidna
+
+We will use a simple example to show how `multi-abi` works. We will be using two contracts, `Flag` and `EchidnaTest`, both available in [`multiabi.sol`](example/multiabi.sol).
+
+The `Flag` contract contains a boolean flag that is only set if `flip()` is called, and a getter function that returns the value of the flag. For now, ignore `test_fail()`, we will talk about this function later.
+
+```solidity
+contract Flag {
+
+   bool flag = false;
+
+   function flip() public {
+       flag = !flag;
+   }
+
+   function get() public returns (bool) {
+        return flag;
+   }
+
+   function test_fail() public {
+       assert(false);
+   }
+}
+```
+
+The test harness will instantiate a new `Flag`, and the invariant under test will be that `f.get()` (that is, the boolean value of the flag) is always false. 
+
+```solidity
+contract EchidnaTest {
+   Flag f;
+
+   constructor() {
+      f = new Flag();
+   }
+
+   function test_flag_is_false() public {
+      assert(f.get() == false);
+   }
+
+}
+```
+
+In a non `multi-abi` fuzzing campaign, Echidna is not able to break the invariant, because it only interacts with `EchidnaTest` functions. However, if we use the following configuration file, enabling `multi-abi` testing, the invariant is broken. You can access [`multiabi.yaml` here](example/multiabi.yaml).
+
+```yaml
+testMode: assertion
+testLimit: 50000
+multi-abi: true
+```
+
+To run the Echidna tests, run `echidna-test multiabi.sol --contract EchidnaTest --config multiabi.yaml` from the `example` directory. Alternatively, you can specify `--multi-abi` in the command line instead of using a configuration file.
+
+### Example run with `multi-abi` set to `false`
+
+```
+$ echidna-test multiabi.sol --contract EchidnaTest --config multiabi.yaml 
+Analyzing contract: building-secure-contracts/program-analysis/echidna/example/multiabi.sol:EchidnaTest
+test_flag_is_false():  passed! 🎉
+AssertionFailed(..):  passed! 🎉
+
+Unique instructions: 282
+Unique codehashes: 2
+Corpus size: 2
+Seed: -8252538430849362039
+```
+
+### Example run with `multi-abi` set to `true`
+
+```
+$ echidna-test multiabi.sol --contract EchidnaTest --config multiabi.yaml 
+Analyzing contract: building-secure-contracts/program-analysis/echidna/example/multiabi.sol:EchidnaTest
+test_flag_is_false(): failed!💥  
+  Call sequence:
+    flip()
+    flip()
+    flip()
+    test_flag_is_false()
+
+Event sequence: Panic(1)
+AssertionFailed(..):  passed! 🎉
+
+Unique instructions: 368
+Unique codehashes: 2
+Corpus size: 6
+Seed: -6168343983565830424
+```
+
+## Use cases and conclusions
+
+Testing with `multi-abi` is a useful tool for complex systems that require the user to interact with more than one contract, as we mentioned earlier. Another use case is for deployed contracts that require interactions to be initiated by specific addresses: for those, specifying the `sender` configuration setting allows to send the transactions from the correct account.
+
+A side-effect of using `multi-abi` is that the search space grows with the number of functions that can be called. This, combined with high values of sequence lengths, can make the fuzzing test not so thorough, because the dimension of the search space is simply too big to reasonably explore. Finally, adding more functions as fuzzing candidates makes the campaigns to take up more execution time.
+
+A final remark is that `multi-abi` testing in assertion mode ignores all assert failures from the contracts not under test. This is shown in `Flag.test_fail()` function: even though it explicitly asserts false, the Echidna test ignores it.