Merge pull request #273 from crytic/dev-ffi-cheatcode

Add FFI cheatcode example

Author: montyly

SUMMARY.md

@@ -82,6 +82,7 @@
       - [How to seed Echidna with unit tests](./program-analysis/echidna/advanced/end-to-end-testing.md)
       - [How to fuzz contracts with external libraries](./program-analysis/echidna/advanced/working-with-libraries.md)
       - [Understanding and using `multi-abi`](./program-analysis/echidna/advanced/using-multi-abi.md)
+      - [Interacting with off-chain data via FFI cheatcode](./program-analysis/echidna/advanced/interacting-with-offchain-data-via-ffi.md)
     - [Fuzzing tips](./program-analysis/echidna/fuzzing_tips.md)
     - [Frequently Asked Questions](./program-analysis/echidna/frequently_asked_questions.md)
     - [Exercises](./program-analysis/echidna/exercises/README.md)

program-analysis/echidna/advanced/README.md

@@ -9,3 +9,4 @@
 - [How to Use Hevm Cheats to Test Permit](./hevm-cheats-to-test-permit.md): Find out how to test code that relies on ecrecover signatures by using hevm cheat codes.
 - [How to Seed Echidna with Unit Tests](./end-to-end-testing.md): Discover how to use existing unit tests to seed Echidna.
 - [Understanding and Using `multi-abi`](./using-multi-abi.md): Learn what `multi-abi` testing is and how to utilize it effectively.
+- [Interacting with off-chain data via FFI cheatcode](./interacting-with-offchain-data-via-ffi.md): Using the `ffi` cheatcode as a way of communicating with the operating system.

program-analysis/echidna/advanced/interacting-with-offchain-data-via-ffi.md

@@ -0,0 +1,104 @@
+# Interacting with off-chain data using the `ffi` cheatcode
+
+## Introduction
+
+It is possible for Echidna to interact with off-chain data by means of the `ffi` cheatcode. This function allows the caller to execute an arbitrary command on the system running Echidna and read its output, enabling the possibility of getting external data into a fuzzing campaign.
+
+## A word of caution
+
+In general, the usage of cheatcodes is not encouraged, since manipulating the EVM execution environment can lead to unpredictable results and false positives or negatives in fuzzing tests.
+
+This piece of advice becomes more critical when using `ffi`. This cheatcode basically allows arbitrary code execution on the host system, so it's not just the EVM execution environment that can be manipulated. Running malicious or untrusted tests with `ffi` can have disastrous consequences.
+
+The usage of this cheatcode should be extremely limited, well documented, and only reserved for cases where there is not a secure alternative.
+
+## Pre-requisites
+
+If reading the previous section didn't scare you enough and you still want to use `ffi`, you will need to explicitly tell Echidna to allow the cheatcode in the tests. This safety measure makes sure you don't accidentally execute `ffi` code.
+
+To enable the cheatcode, set the `allowFFI` flag to `true` in your Echidna configuration file:
+
+```yaml
+allowFFI: true
+```
+
+## Uses
+
+Some of the use cases for `ffi` are:
+
+- Making prices or other information available on-chain during a fuzzing campaign. For example, you can use `ffi` to feed an oracle with "live" data.
+- Get randomness in a test. As you know, there is no randomness source on-chain, so using this cheatcode you can get a random value from the device running the fuzz tests.
+- Integrate with algorithms not ported to Solidity language, or perform comparisons between two implementations. Some examples for this item include signing and hashing, or custom calculations algorithms.
+
+## Example: Call an off-chain program and read its output
+
+This example will show how to create a simple call to an external executable, passing some values as parameters, and read its output. Keep in mind that the return values of the called program should be an abi-encoded data chunk that can be later decoded via `abi.decode()`. No newlines are allowed in the return values.
+
+Before digging into the example, there's something else to keep in mind: When interacting with external processes, you will need to convert from Solidity data types to string, to pass values as arguments to the off-chain executable. You can use the [crytic/properties](https://github.com/crytic/properties) `toString` [helpers](https://github.com/crytic/properties/blob/main/contracts/util/PropertiesHelper.sol#L447) for converting.
+
+For the example we will be creating a python example script that returns a random `uint256` value and a `bytes32` hash calculated from an integer input value. This doesn't represent a "useful" use case, but will be enough to show how the `ffi` cheatcode is used. Finally, we won't perform sanity checks for data types or values, we will just assume the input data will be correct.
+
+This script was tested with Python 3.11, Web3 6.0.0 and eth-abi 4.0.0. Some functions had different names in prior versions of the libraries.
+
+```python
+import sys
+import secrets
+from web3 import Web3
+from eth_abi import encode
+
+# Usage: python3 script.py number
+number = int(sys.argv[1])
+
+# Generate a 10-byte random number
+random = int(secrets.token_hex(10), 16)
+
+# Generate the keccak hash of the input value
+hashed = Web3.solidity_keccak(['uint256'], [number])
+
+# ABI-encode the output
+abi_encoded = encode(['uint256', 'bytes32'], [random, hashed]).hex()
+
+# Make sure that it doesn't print a newline character
+print("0x" + abi_encoded, end="")
+```
+
+You can test this program with various inputs and see what the output is. If it works correctly, the program should output a 512-bit hex string that is the ABI-encoded representation of a 256-bit integer followed by a bytes32.
+
+Now let's create the Solidity contract that will be run by Echidna to interact with the previous script.
+
+```solidity
+pragma solidity ^0.8.0;
+
+// HEVM helper
+import "@crytic/properties/contracts/util/Hevm.sol";
+
+// Helpers to convert uint256 to string
+import "@crytic/properties/contracts/util/PropertiesHelper.sol";
+
+contract TestFFI {
+    function test_ffi(uint256 number) public {
+        // Prepare the array of executable and parameters
+        string[] memory inp = new string[](3);
+        inp[0] = "python3";
+        inp[1] = "script.py";
+        inp[2] = PropertiesLibString.toString(number);
+
+        // Call the program outside the EVM environment
+        bytes memory res = hevm.ffi(inp);
+
+        // Decode the return values
+        (uint256 random, bytes32 hashed) = abi.decode(res, (uint256, bytes32));
+
+        // Make sure the return value is the expected
+        bytes32 hashed_solidity = keccak256(abi.encodePacked(number));
+        assert(hashed_solidity == hashed);
+    }
+}
+```
+
+The minimal configuration file for this test is the following:
+
+```yaml
+testMode: "assertion"
+allowFFI: true
+```