diff --git a/.gitbook/assets/contract-folder.png b/.gitbook/assets/contract-folder.png index 223f52b..7a14cf4 100644 Binary files a/.gitbook/assets/contract-folder.png and b/.gitbook/assets/contract-folder.png differ diff --git a/.gitbook/assets/core-folder.png b/.gitbook/assets/core-folder.png index fd5ea13..58a3aa1 100644 Binary files a/.gitbook/assets/core-folder.png and b/.gitbook/assets/core-folder.png differ diff --git a/.gitbook/assets/utils-folder.png b/.gitbook/assets/utils-folder.png index 87e422d..784d508 100644 Binary files a/.gitbook/assets/utils-folder.png and b/.gitbook/assets/utils-folder.png differ diff --git a/bdk-implementation/README.md b/bdk-implementation/README.md index 0c45002..2a88614 100644 --- a/bdk-implementation/README.md +++ b/bdk-implementation/README.md @@ -6,25 +6,34 @@ description: How the functional elements of AppLayer interact with each other. This chapter aims to explain in technical detail how the BDK is implemented, as well as its submodules and how everything comes together to deliver a blazing fast blockchain. -The first few subchapters paint a more holistic view of the BDK, as most components are pretty straight-forward to understand, and developers are expected to use the [Doxygen](https://doxygen.nl) documentation as a reference to further understand how the project works. The later subchapters show some components that are particularly denser and/or complex enough that they warrant their own separated explanations. +The first few subchapters paint a more holistic view of the BDK, as most components are pretty straight-forward to understand. Developers are expected to use the [Doxygen](https://doxygen.nl) documentation as a reference to further understand how the project works. The later subchapters show some components that are particularly denser and/or complex enough that they warrant their own separated explanations. Looking at a higher level of abstraction, the original C++ implementation of the BDK is structured like this: -* The `src/bins` folder contains the source files for the project's main executables - the blockchain executable itself, contract ABI generator and other testing-related executables are all coded here in their respective subfolders +* The `src/bins` folder contains the source files for the project's main executables - the blockchain executable itself, contract ABI generator, network simulator, faucet API and other testing-related executables are all coded here in their respective subfolders +* The `src/bytes` folder contains code related to the `bytes` class, a container that deals with raw bytes - specifically the `bytes::join()` function and the `bytes::View` class, both used extensively across the project * The `src/contract` folder contains everything related to the logic of smart contracts - from ABI parsing to custom variable types and template contracts * The `src/core` folder contains the heart of the BDK - the main components of the blockchain and what makes it tick * The `src/libs` folder contains third-party libraries not inherently tied to the project but used throughout development * The `src/net` folder contains everything related to networking, communication between nodes and support for protocols such as gRPC, HTTP, P2P, JSON-RPC, etc. * The `src/utils` folder contains several commonly-used functions, structures, classes and overall logic to support the functioning of the BDK as a whole +There is also a `tests` folder that contains several unit tests for each of the components described above. + ## Source tree For the more visually inclined, here is a source tree (headers only) containing all of the files inside the `src` folder (except `src/bins` as it only contains source files), their respective subfolders and which components are declared in them. Each component is further explained through the following subchapters of this documentation. For more technical details (e.g. API references for developers), please refer to the [Doxygen](https://www.doxygen.nl) documentation on the project's own repository. ``` src +├── bytes +│   ├── initializer.h (bytes::Initializer, bytes::SizedInitializer) +│   ├── join.h (bytes::join()) +│   ├── range.h (bytes::Range, bytes::DataRange, bytes::BorrowedDataRange) +│   └── view.h (bytes::View, bytes::Span) ├── contract (Contracts) │   ├── abi.h (ABI - encoders, decoders, helper structs, etc.) +│   ├── calltracer.h (trace namespace, Call struct, CallTracer class) │   ├── contractfactory.h (ContractFactory) │   ├── contract.h (ContractGlobals, ContractLocals, BaseContract) │   ├── contracthost.h (ContractHost) @@ -32,7 +41,7 @@ src │   ├── contractstack.h (ContractStack) │   ├── customcontracts.h (for declaring custom contracts) │   ├── dynamiccontract.h (DynamicContract) -│   ├── event.h (Event, EventManager) +│   ├── event.h (Event) │   ├── templates (folder for contract templates) │   │ ├── dexv2 (subfolder for the DEXV2 contract components) │   │ │ ├── dexv2factory.h (DEXV2Factory) @@ -44,37 +53,42 @@ src │   │ ├── erc20wrapper.h (ERC20Wrapper) │   │ ├── erc721.h (ERC721) │   │ ├── erc721test.h (ERC721Test, used solely for testing purposes) +│   │   ├── erc721uristorage.h (ERC721URIStorage, converted from OpenZeppelin) │   │ ├── nativewrapper.h (NativeWrapper) +│   │   ├── ownable.h (Ownable, converted from OpenZeppelin) +│   │   ├── pebble.h (Pebble) │   │ ├── randomnesstest.h (RandomnessTest) │   │ ├── simplecontract.h (SimpleContract) +│   │   ├── snailtracer.h, snailtraceroptimized.h (SnailTracer and SnailTracerOptimized, converted from the original EVM impl) │   │ ├── testThrowVars.h (TestThrowVars, used solely for testing purposes) -│   │ └── throwtestA.h, throwtestB.h, throwtestC.h (for testing CM nested calls) +│   │ └── throwtestA.h, throwtestB.h, throwtestC.h (for testing nested contract calls) │   └── variables (Safe Variables for use within Dynamic Contracts) │   ├── reentrancyguard.h (ReentrancyGuard) │   ├── safeaddress.h (SafeAddress) │   ├── safearray.h (SafeArray) │   ├── safebase.h (SafeBase - used as base for all other types) │   ├── safebool.h (SafeBool) -│   ├── safeint.h (SafeInt) +│   ├── safebytes.h (SafeBytes) +│   ├── safeint.h (SafeInt and respective aliases) │   ├── safestring.h (SafeString) │   ├── safetuple.h (SafeTuple) -│   ├── safeuint.h (SafeUint) +│   ├── safeuint.h (SafeUint and respective aliases) │   ├── safeunorderedmap.h (SafeUnorderedMap) │   └── safevector.h (SafeVector) ├── core (Core components) │   ├── blockchain.h (Blockchain, Syncer) │   ├── consensus.h (Consensus) │   ├── dump.h (Dumpable, DumpManager, DumpWorker) -│   ├── rdpos.h (Validator, rdPoS, rdPoSWorker) -│   ├── state.h (State) +│   ├── rdpos.h (Validator, rdPoS) +│   ├── state.h (BlockValidationStatus, State) │   └── storage.h (Storage) -├── libs (Third-party libs) +├── libs (Third-party libraries) │ ├── BS_thread_pool_light.hpp (https://github.com/bshoshany/thread-pool) │ ├── catch2/catch_amalgamated.hpp (https://github.com/catchorg/Catch2) -│ └── json.hpp (https://github.com/nlohmann/json) +│ ├── json.hpp (https://github.com/nlohmann/json) +│ ├── wyhash.h (https://github.com/wangyi-fudan/wyhash) +│ └── zpp_bits.h (https://github.com/eyalz800/zpp_bits) ├── net (Networking) -│   ├── grpcclient.h (gRPCClient) -│   ├── grpcserver.h (gRPCServer) │   ├── http (HTTP part of networking) │   │   ├── httpclient.h (HTTPClient) │   │   ├── httplistener.h (HTTPListener) @@ -82,33 +96,40 @@ src │   │   ├── httpserver.h (HTTPServer) │   │   ├── httpsession.h (HTTPQueue, HTTPSession) │   │   └── jsonrpc (Namespace for handling JSONRPC data) -│   │   ├── decoding.h (functions for decoding JSONRPC data) -│   │   ├── encoding.h (functions for encoding JSONRPC data) -│   │   └── methods.h (declarations for JSONRPC methods) +│   │   ├── blocktag.h (BlockTag, BlockTagOrNumber) +│   │   ├── call.h (call() - a function that processes a JSON RPC call) +│   │   ├── error.h (Error - for abstracting JSON RPC errors) +│   │   ├── methods.h (contains all implemented JSON RPC methods) +│   │   ├── parser.h (Parser and helper tempate functions) +│   │   └── variadicparser.h (VariadicParser and helper template functions) │   └── p2p (P2P part of networking) -│   ├── client.h (ClientFactory) +│   ├── broadcaster.h (Broadcaster) │   ├── discovery.h (DiscoveryWorker - worker thread for ManagerDiscovery) │   ├── encoding.h (collection of enums, structs, classes, encoders and decoders used in P2P communications) │   ├── managerbase.h (ManagerBase - used as base for ManagerDiscovery and ManagerNormal) │   ├── managerdiscovery.h (ManagerDiscovery) │   ├── managernormal.h (ManagerNormal) │   ├── nodeconns.h (NodeConns) -│   ├── server.h (ServerListener, Server) │   └── session.h (Session) -└── utils (Base components) - ├── contractreflectioninterface.h (ContractReflectionInterface - interface for registering contracts) +└── utils (Utility components) + ├── clargs.h (definitions for helper functions, enums and structs that deal with command-line argument parsing) + ├── contractreflectioninterface.h (ContractReflectionInterface - interface for registering Dynamic Contracts) ├── db.h (DBPrefix, DBServer, DBEntry, DBBatch, DB) ├── dynamicexception.h (DynamicException - custom exception class) ├── ecdsa.h (PrivKey, Pubkey, UPubkey, Secp256k1) + ├── evmcconv.h (EVMCConv - namespace for EVMC-related data conversion functions) ├── finalizedblock.h (FinalizedBlock) ├── hex.h (Hex) + ├── intconv.h (IntConv - namespace for signed integer aliases and data conversion functions) ├── jsonabi.h (JsonAbi - namespace for writing contract ABIs to JSON format) - ├── logger.h (LogType, Log, LogInfo, Logger) + ├── logger.h (LogType, Log, LogInfo, Logger, LogicalLocationProvider) ├── merkle.h (Merkle) ├── options.h (Options singleton - generated by CMake through a .in file) ├── randomgen.h (RandomGen) ├── safehash.h (SafeHash, FNVHash) - ├── strings.h (FixedBytes, Hash, Functor, Signature, Address) + ├── strconv.h (StrConv - namespace for string-related data conversion and manipulation functions) + ├── strings.h (FixedBytes and its derivatives - Hash, Functor, Signature, Address, StorageKey) ├── tx.h (TxBlock, TxValidator) - └── utils.h (definitions for Bytes/Account structs, uint/ethCallInfo types, Networks, and the Utils namespace) + ├── uintconv.h (UintConv - namespace for unsigned integer aliases and data conversion functions) + └── utils.h (Utils namespace and other misc struct and enum definitions) ``` diff --git a/bdk-implementation/contract-call-handling.md b/bdk-implementation/contract-call-handling.md index 4854f9d..1770804 100644 --- a/bdk-implementation/contract-call-handling.md +++ b/bdk-implementation/contract-call-handling.md @@ -12,69 +12,42 @@ For example, a call with a variable that starts with the value "10", then five c ## ContractStack class overview -Here's an overview of the `ContractStack` class definition and functionalities: +Here's an overview of the `ContractStack` class definition and functionalities (comments removed for easier reading, check the `contract/contractstack.h` file for more details): ```c++ class ContractStack { private: - std::unordered_map code_; - std::unordered_map balance_; - std::unordered_map nonce_; - std::unordered_map storage_; + boost::unordered_flat_map code_; + boost::unordered_flat_map balance_; + boost::unordered_flat_map nonce_; + boost::unordered_flat_map storage_; std::vector events_; - std::vector
contracts_; // Contracts that have been created during the execution of the call, we need to revert them if the call reverts. + std::vector> contracts_; std::vector> usedVars_; public: - ContractStack() = default; - ~ContractStack() = default; - - inline void registerCode(const Address& addr, const Bytes& code) { - if (!this->code_.contains(addr)) { - this->code_[addr] = code; - } - } - - inline void registerBalance(const Address& addr, const uint256_t& balance) { - if (!this->balance_.contains(addr)) { - this->balance_[addr] = balance; - } - } - - inline void registerNonce(const Address& addr, const uint64_t& nonce) { - if (!this->nonce_.contains(addr)) { - this->nonce_[addr] = nonce; - } - } - - inline void registerStorageChange(const StorageKey& key, const Hash& value) { - if (!this->storage_.contains(key)) { - this->storage_[key] = value; - } - } - - inline void registerEvent(Event&& event) { - this->events_.emplace_back(std::move(event)); - } - - inline void registerContract(const Address& addr) { - this->contracts_.push_back(addr); - } - - inline void registerVariableUse(SafeBase& var) { - this->usedVars_.emplace_back(var); - } - - /// Getters - inline const std::unordered_map& getCode() const { return this->code_; } - inline const std::unordered_map& getBalance() const { return this->balance_; } - inline const std::unordered_map& getNonce() const { return this->nonce_; } - inline const std::unordered_map& getStorage() const { return this->storage_; } + inline void registerCode(const Address& addr, const Bytes& code) { this->code_.try_emplace(addr, code); } + + inline void registerBalance(const Address& addr, const uint256_t& balance) { this->balance_.try_emplace(addr, balance); } + + inline void registerNonce(const Address& addr, const uint64_t& nonce) { this->nonce_.try_emplace(addr, nonce); } + + inline void registerStorageChange(const StorageKey& key, const Hash& value) { this->storage_.try_emplace(key, value); } + + inline void registerEvent(Event event) { this->events_.emplace_back(std::move(event)); } + + inline void registerContract(const Address& addr, BaseContract* contract) { this->contracts_.emplace_back(addr, contract); } + + inline void registerVariableUse(SafeBase& var) { this->usedVars_.emplace_back(var); } + + inline const boost::unordered_flat_map& getCode() const { return this->code_; } + inline const boost::unordered_flat_map& getBalance() const { return this->balance_; } + inline const boost::unordered_flat_map& getNonce() const { return this->nonce_; } + inline const boost::unordered_flat_map& getStorage() const { return this->storage_; } inline std::vector& getEvents() { return this->events_; } - inline const std::vector
& getContracts() const { return this->contracts_; } + inline const std::vector>& getContracts() const { return this->contracts_; } inline const std::vector>& getUsedVars() const { return this->usedVars_; } }; ``` -The existence of only *one* instance of `ContractStack` per `ContractHost`, and its integration within the RAII framework of `ContractHost`, guarantees that state values are meticulously committed or reverted upon the completion or rollback of transactions. This robust design prevents state spill-over between different contract executions, fortifying transaction isolation and integrity across the blockchain network - even in the dynamic and mutable landscape of blockchain transactions, the integrity and consistency of state changes are meticulously maintained, safeguarding against unintended consequences and errors during contract execution. - +The existence of only *one* instance of `ContractStack` per `ContractHost`, as well as its integration within the RAII framework of `ContractHost`, guarantees that state values are meticulously committed or reverted upon the completion or rollback of transactions. This robust design prevents state spill-over between different contract executions, fortifying transaction isolation and integrity across the blockchain network - even in the dynamic and mutable landscape of blockchain transactions, the integrity and consistency of state changes are meticulously maintained, safeguarding against unintended consequences and errors during contract execution. diff --git a/bdk-implementation/database.md b/bdk-implementation/database.md index e112682..5229a30 100644 --- a/bdk-implementation/database.md +++ b/bdk-implementation/database.md @@ -8,11 +8,17 @@ BDK validators use an in-disk database for storing data about themselves and oth The database itself is an abstraction of a [Speedb](https://github.com/speedb-io/speedb) database - a simple key/value database, but handled in a different way: keys use *prefixes*, which makes it possible to batch read and write, so we can get around the "simple key/value" limitation and divide data into sectors. -The database requires a filesystem path to open it (if it already exists) or create it on the spot (if it doesn't exist) during construction. It closes itself automatically on destruction. +The database requires a filesystem path to open it (if it already exists) or create it on the spot (if it doesn't exist) during construction. It closes itself automatically on destruction. Optionally, it also accepts a bool for enabling compression (disabled by default), if needed. Content in the database is stored as raw bytes. This is due to space optimizations, as one raw byte equals two UTF-8 characters (e.g. an address like `0x1234567890123456789012345678901234567890`, ignoring the "0x" prefix, occupies 20 raw bytes - "12 34 56 ..." - , but 40 bytes if converted to a string, since each byte becomes two separate characters - "1 2 3 4 5 6 ..."). -For the main CRUD operations, refer to the `has()`, `get()`, `put()` and `del()` functions. Due to how the database works internally, updating an entry is the same as inserting a different value in a key that already exists, effectively replacing the value that existed before (e.g. `put(oldKey, newValue)`). There's also `getBatch()` and `putBatch()` for batched operations, as well as `getKeys()` for fetching only the keys. +For the main CRUD operations, refer to the `has()`, `get()`, `put()` and `del()` functions. Due to how the database works internally, updating an entry is the same as inserting a different value in a key that already exists, effectively replacing the value that existed before (e.g. `put(oldKey, newValue)`). There's also a few other helper functions such as: + +* `getBatch()` and `putBatch()` for batched operations +* `getKeys()` for fetching only the database's keys +* `keyFromStr()` for encapsulating a key into a Bytes object +* `getLastByPrefix()` for getting the last value stored in a given prefix +* `makeNewPrefix()` for concatenating prefixes when necessary ## Structs and Prefixes @@ -24,18 +30,19 @@ We have three helper structs to ease database manipulation: We also have a `DBPrefix` namespace to reference the database's prefixes in a simpler way: -| Descriptor | Prefix | -| --------------- | ------ | -| blocks | 0x0001 | -| blockHeightMaps | 0x0002 | -| nativeAccounts | 0x0003 | -| txToBlocks | 0x0004 | -| rdPoS | 0x0005 | -| contracts | 0x0006 | -| contractManager | 0x0007 | -| events | 0x0008 | -| vmStorage | 0x0009 | -| txToAddr | 0x000A | +| Descriptor | Prefix | +| ------------------ | ------ | +| blocks | 0x0001 | +| heightToBlock | 0x0002 | +| nativeAccounts | 0x0003 | +| txToBlock | 0x0004 | +| rdPoS | 0x0005 | +| contracts | 0x0006 | +| contractManager | 0x0007 | +| events | 0x0008 | +| vmStorage | 0x0009 | +| txToAdditionalData | 0x000A | +| txToCallTrace | 0x000B | Those prefixes are concatenated to the start of the _key_, so an entry that would have, for example, a key named "abc" and a value of "123", if inserted to the "0003" prefix, would be like this inside the database (in raw bytes format, strings here are just for the sake of the explanation): `{"0003abc": "123"}` @@ -49,7 +56,7 @@ Used to store serialized blocks based on their hashes. | ------------------ | ---------------- | | Prefix + BlockHash | Serialized Block | -### blockHeightMaps +### heightToBlock Used to store block hashes based on their heights. @@ -73,7 +80,7 @@ For example, an account with balance 1000000 and nonce 2 would be serialized as An account with balance 0 and nonce 0 would be serialized as `0000`. -### txToBlocks +### txToBlock Used to store block hashes, the tx indexes within that block and the block heights, based on their transaction hashes. @@ -132,11 +139,18 @@ Used to store EVM-related stuff like storage keys and values (essentially the EV | ------------------------------------------- | ------------------------ | | Address (20 bytes) + Storage Key (32 bytes) | Storage Value (32 bytes) | -### txToAddr +### txToAdditionalData + +Used to store EVM transactions that created contracts, storing the transaction hash and additional data about the contract that was deployed (see the `TxAdditionalData` struct in `utils/tx.h` for more details). + +| Key | Value | +| ------------------------ | ---------------------------------- | +| Prefix + TransactionHash | Serialized TxAdditionalData struct | -Used to store EVM transactions that created contracts, storing the transaction hash and the address where the contract was deployed. +### txToCallTrace -| Key | Value | -| ------------------------ | ------------------ | -| Prefix + TransactionHash | ContractAddress | +Used to store debugging information about contract calls - see the `Call` struct in `contract/calltracer.h` for more details. +| Key | Value | +| ------------------------ | ---------------------- | +| Prefix + TransactionHash | Serialized Call struct | diff --git a/bdk-implementation/p2p-encoding.md b/bdk-implementation/p2p-encoding.md index fc5d8c5..69b36ce 100644 --- a/bdk-implementation/p2p-encoding.md +++ b/bdk-implementation/p2p-encoding.md @@ -137,7 +137,7 @@ Broadcast a given TxValidator to all known connected nodes connected. They shoul Broadcast a given TxBlock to all known connected nodes. They should validate and rebroadcast. * Command ID: `0005` -* Request Type: Broadcast only +* Request Type: Broadcast * Request Payload: Tx RLP * Request Example: `0x02adf01827349cad810005f86b02851087ee060082520894f137c97b1345f0a7ec97d070c70cf96a3d71a1c9871a204f293018008025a0d738fcbf48d672da303e56192898a36400da52f26932dfe67b459238ac86b551a00a60deb51469ae5b0dc4a9dd702bad367d1111873734637d428626640bcef15c` @@ -146,7 +146,7 @@ Broadcast a given TxBlock to all known connected nodes. They should validate and Broadcast a given Block to all known connected nodes. They should validate and rebroadcast. * Command ID: `0006` -* Request Type: Broadcast only +* Request Type: Broadcast * Request Payload: Block RLP * Request Example: `0x02adf01827349cad81000618395ff0c8ee38a250b9e7aeb5733c437fed8d6ca2135fa634367bb288a3830a3c624e33401a1798ce09f049fb6507adc52b085d0a83dacc43adfa519c1228e70122143e16db549af9ccfd3b746ea4a74421847fa0fe7e0e278626a4e7307ac0f600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000186c872a48300000000057de95400000000000000d918395ff0c8ee38a250b9e7aeb5733c437fed8d6ca2135fa634367bb288a3830a3c624e33401a1798ce09f049fb6507adc52b085d0a83dacc43adfa519c1228e70122143e16db549af9ccfd3b746ea4a74421847fa0fe7e0e278626a4e7307ac0f600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000186c872a48300000000057de95400000000000000d9` @@ -163,3 +163,29 @@ Request the TxBlock memory pool of another node. * To encode this, the node has to encode the number of transactions (`0x00000002`), and then append the transactions' sizes and datas in an ordered manner * The resulting answer message would be `0x01adf01827349cad810003000000020000006df86ba4cfffe74621e0caa14dfcb67a6f263391f4a6ad957ace256f3e706b6da07517a63295165701823f44a08cfdf3826149ca0317eaf4f5832c867a4f5050e3e70d635323947d61a4f35618a07dbe86e6a8cef509c7c6174cb8c703ddd8cb695511d72043630d99888ff2ba200000006df86ba4cfffe74621e0caa14dfcb67a6f263391f4a6ad957ace256f3e706b6da07517a63295165701823f44a01545c0c89ad5fda9e4c6ef65f264ef575fa2edebef29d577f88d365ff9d28357a00d9ed64e1675315477aca44908148b9b572c7de45d420398193dcfc2d430d158` +### NotifyInfo + +Same as Info, but as a notification instead (doesn't request the other node's info). + +* Command ID: `0008` +* Request Type: Notification +* Request Payload: node version + epoch + latest nHeight + latest nHash + nConnectedNodes +* Request Example: `0x03adf01827349cad81000800000000000000010005f70436085980000000000000000156aef6ce3d9cefb653ea6a53fc8e810c95268c01428223a8ee267ed2ac9f05d8` + +| Variable | Size | Info | +| -------- | -------- | ------------------------------ | +| version | 8 Bytes | Node Version | +| Epoch | 8 Bytes | Timestamp in UNIX microseconds | +| nHeight | 8 Bytes | latest block nHeight | +| Hash | 32 Bytes | latest block hash | + +### RequestBlock + +Requests data from a given block or range of blocks to another node. + +* Command ID: `0009` +* Request Type: Answer or Request +* Request Payload: block height start range + block height end range + bytes limit (minimum 1 block) +* Request Example: `0x00d6906a45be95f8cb0009000000000000000000000000000000010000000000000400` +* Answer Payload: serialized block size + content (1..N blocks) +* Answer Example: `0x0164fca114cca6773b0009000000000000346bc95fe4d72f96048212a311e3ccfb3ceaef75e801fe1e84ada2b459e185ba097d0871716384a2b3c7076f380f7873d6251e38af74e916fa8eaa0e3ca65347b2c9010db6bd216c7910cccb1dea09f6e7a1c1488446ebacb82cb3ef46c2341b59b0eacfd30704b89f92a41ae8970dd2c001eb08273a5d3c5a7beede721dcd6bcf5ad89c329d6db3658c6da18743131bc95116819fdb8ea16b0abdb7eaf19c840ed09f3e3136569f2d6edab18568da6aef33ae18b8b02ad0a998af0364e184fa987236000626f79d95b771000000000000000a00000000000030e300000077...` diff --git a/bdk-implementation/p2p-overview.md b/bdk-implementation/p2p-overview.md index 84ddeee..be3836f 100644 --- a/bdk-implementation/p2p-overview.md +++ b/bdk-implementation/p2p-overview.md @@ -6,9 +6,13 @@ description: A primer on how P2P messaging works in the Blockchain Development K This subchapter provides a comprehensive overview of the P2P classes and their organization within the BDK. It further elaborates on the life-cycle of a P2P connection and the dynamic flow of data between nodes. All classes described here are inside the `src/net/p2p` folder. Check their header files for full details on how each of them work. +## NodeConns + +The `NodeConns` class (`nodeconns.h`) takes the role of maintaining and periodically updating a list of all peer nodes in the network that are connected to the local node, as well as their respective info. It is through this list that the node keeps itself up-to-date with the most recent node info possible, while also dropping stale connections/timed out peers every now and then to avoid potential network sync problems. + ## Session -The `Session` class (`session.h`) encapsulates a TCP connection with a remote node, responsible for managing handshakes, sending and receiving messages by reading and writing socket data. It has a queue for outbound messages, allowing any thread that is responsible for sending a message to carry on with its task without having to wait for the message transmission to complete, and serves as a base for the `ClientSession` and `ServerSession` specialized classes. +The `Session` class (`session.h`) encapsulates a TCP connection with a remote node, responsible for managing handshakes, sending and receiving messages by reading and writing socket data. It servees for both client and server connections, having queues for both inbound and outbound messages, allowing any thread that is responsible for sending a message to carry on with its task without having to wait for the message transmission to complete. Sessions are designed to be used as `shared_ptr`s with `shared_from_this` to manage their lifecycles, as they primarily exist within the handlers of Boost's `io_context`. Once a session has successfully established a connection with a remote node, it is added to a list of sessions controlled by a manager. It is critical to properly manage those shared pointers, as the destructor of the `Session` class is intrinsically linked to the destructor of the socket, which, in turn, calls the `io_context`. If the pointer persists but its referenced `io_context` no longer exists, this will lead to a program crash. @@ -16,7 +20,7 @@ For instance, functions called by the thread pool to parse asynchronous operatio ## Life cycle of a session -Upon instantiation, a session's life-cycle is composed of three different routines: _handshake_, _read_ and _write_. +Upon instantiation, a session's life-cycle is composed of three different routines: *handshake*, *read* and *write*. ### Handshake routine @@ -49,31 +53,9 @@ The third routine is writing outbound messages going to the remote endpoint, lik * `do_write_message()` writes the full message, and calls `on_write_message()` when done * `on_write_message()` is a callback for when the message write process is done, locking the queue mutex, grabbing the next message from the queue, and calling `do_write_header()` again - this loop goes on until the queue is empty, where the outbound message pointer is set to null and any further callbacks from the write strand come to a halt -## ClientFactory - -The `ClientFactory` class (`client.h`) serves as an intermediary that allows the P2P manager to instantiate and launch a new outbound client session with a given IP address and port. It does not directly manage these instances, but rather facilitates their creation and execution within an `io_context`. At present, the class operates the `io_context` across _four_ threads. We use strands to effectively manage and execute asynchronous operations within the sessions. - -The code snippet below, demonstrates how a new `ClientSession` is instantiated and put into operation by `ClientFactory`: - -```cpp -void ClientFactory::createClientSession(const boost::asio::ip::address &address, const unsigned short &port) { - tcp::socket socket(this->io_context_); - auto session = std::make_shared(std::move(socket), ConnectionType::OUTBOUND, manager_, this->threadPool_, address, port); - session->run(); -} - -void ClientFactory::connectToServer(const boost::asio::ip::address &address, const unsigned short &port) { - boost::asio::post(this->connectorStrand_, std::bind(&ClientFactory::createClientSession, this, address, port)); -} -``` - -## Server - -The `Server` class (`server.h`) acts as a straightforward TCP server, listening for and creating new sessions for each incoming connection. It does not directly manage the sessions, it only focuses on accepting connections, creating corresponding sessions, and executing them within a designated `io_context`. - ## ManagerBase and derivatives -The P2P manager acts as the backbone of the P2P network. It bears the responsibility of managing `Session`s, the `Server`, and the `ClientFactory`, overseeing their operations and indirectly managing the `io_context`. Just like the `Session` class and its derivatives, the `ManagerBase` class (`managerbase.h`) serves as a base for the `ManagerNormal` (for Normal nodes) and `ManagerDiscovery` (for Discovery nodes) specialized classes. +The P2P manager acts as the backbone of the P2P network. It bears the responsibility of managing `Session`s, overseeing their operations and indirectly managing the `io_context`, and serves as a base for the `ManagerNormal` (for Normal nodes) and `ManagerDiscovery` (for Discovery nodes) specialized classes. Once a `Session` has successfully completed a handshake, it is registered within the manager, which then oversees the `Session`'s lifecycle. The manager's responsibilities include maintaining a registry of active `Session`s, handling incoming and outgoing requests and responses, and maintaining the communication between them. @@ -83,15 +65,22 @@ It's also important to be aware of the lifespan of the `io_context` and the obje ## Message types -Every incoming message is promptly parsed by the manager via the `handleMessage()` function. These messages can fall into one of three categories: `Request`, `Response`, or `Broadcast`, and each type is handled distinctly by the manager. +Every incoming message is promptly parsed by the manager via the `handleMessage()` function. These messages can fall into one of the following categories, with each one of them being treated distinctly (the names here are conceptual, the actual variable names differ in implementation): -A `Request` message represents a query for specific data from another node (e.g. a list of blocks, a list of transactions, info about the node itself, etc.), while a `Response` message represents an answer to said query. Both types work together in a bidirectional flow that goes like this: +* `Request` - a query for specific data from another node - e.g. a list of blocks, a list of transactions, info about the node itself, etc. +* `Answer` - an answer to a given request +* `Broadcast` - a dissemination of specific data to the network *with* possible re-broadcasting ("flooding"), such as a new block or transaction +* `Notification` - a dissemination of specific data to the network *without* re-broadcasting, such as node info + +Both `Request` and `Answer` messages work together in a bidirectional flow that goes like this: * The sender node initiates a `Request` by generating a random 8-byte ID, registering it internally and sending it alongside the message -* The receiver node receives the `Request`, its manager processes it by invoking the corresponding function to address it, formulates a `Response` with the requested data, assigns it the same ID and sends it back to the sender node -* The sender node receives the `Response` and checks if the received ID is the same one that was registered earlier. If it is, the manager fulfills the associated `Request` future with the received `Response` and deregisters the ID. If the ID is _not_ registered, the `Response` is discarded altogether +* The receiver node receives the `Request`, its manager processes it by invoking the corresponding function to address it, formulates an `Answer` with the requested data, assigns it the same ID and sends it back to the sender node +* The sender node receives the `Answer` and checks if the received ID is the same one that was registered earlier. If it is, the manager fulfills the associated `Request` future with the received `Answer` and deregisters the ID. If the ID is _not_ registered, the `Answer` is discarded altogether + +Both `Notification` and `Broadcast` messages, however, work with a simpler unidirectional flow, as the receiver node doesn't have to answer back to the sender. Instead, it verifies the received data and adds it to its own blockchain. The difference between both types is that `Notification` messages are never re-broadcast, while `Broadcast` messages may or may not be re-broadcast to other nodes, depending on whether said nodes had already received or not said broadcast in the past. -A `Broadcast` message signals a unilateral dissemination of certain data such as a new block or transaction. This type of communication contrasts the `Request`<->`Response` bidirectional flow, as the receiver node doesn't have to answer back to the sender. Instead, it verifies the received data and adds it to its own blockchain, rebroadcasting it to other nodes if necessary. +Due to this specific condition, `Broadcast` messages are specifically handled by the `Broadcaster` class (`net/p2p/broadcaster.h`), while the other types are handled by `ManagerBase` and its derivative classes. ## Asynchronous Message Parsing diff --git a/bdk-implementation/the-contract-folder.md b/bdk-implementation/the-contract-folder.md index 755bcf1..4064d3f 100644 --- a/bdk-implementation/the-contract-folder.md +++ b/bdk-implementation/the-contract-folder.md @@ -12,6 +12,10 @@ This subchapter contains a brief overview of each one of the components inside t The `abi.h` file contains the **ABI** namespace - helper functions for handling Solidity ABI types and data natively, as well as the encoding and decoding of said data, also used for C++ <-> EVM inter-operability. +## CallTracer + +The `calltracer.h` file contains the **Call** struct and the **CallTracer** class respectively, both under the **trace** namespace - those are used specifically for debugging purposes, tracing the flux of a contract call and collecting specific details about it, such as the call type, the sender and receiver addresses, the value sent, the gas used, inputs and outputs, errors and any related subcalls. + ## BaseContract The `contract.h` file contains the **BaseContract** class - the base from which all smart contracts are derived - as well as the **ContractGlobals** and **ContractLocals** helper classes that provide access to global and local variables, respectively, for those contracts to work. @@ -40,11 +44,9 @@ The `customcontracts.h` file contains a tuple that holds all the registered cont The `dynamiccontract.h` file contains the **DynamicContract** class - the base from which all Dynamic Contracts are derived (while _BaseContract_ is mainly used for Protocol Contracts). -## Event and EventManager - -The `event.h` file contains the **Event** and **EventManager** classes, as well as the **EventContainer** typedef for indexing - they represent the subset of contract functionality related to Solidity events. +## Event -_Event_ is responsible for abstracting a Solidity event's structure and data, while _EventManager_ is responsible for maintaining all emitted events within the blockchain. Events are supported for both C++ and EVM contracts. +The `event.h` file contains the **Event** class - an abstraction of a Solidity event's structure and data, used extensively by contracts. Events are managed within the blockchain by the Storage class and are supported for both C++ and EVM contracts. ## The variables subfolder diff --git a/bdk-implementation/the-core-folder.md b/bdk-implementation/the-core-folder.md index 102a3e5..b1ca2be 100644 --- a/bdk-implementation/the-core-folder.md +++ b/bdk-implementation/the-core-folder.md @@ -15,11 +15,12 @@ The `blockchain.h` file contains the **Blockchain** and **Syncer** classes. The **Blockchain** class acts as the mother class that unites all the other components described throughout the docs, including the Syncer. Think of it as "the power button on AppLayer's PC case" - its objective is to be the entry point of the system and act as a mediator for the other components, passing around data to each other, such as (but not limited to): * The global options singleton +* The P2P connection manager * The database * The blockchain history/storage (for blocks and contract events) -* The blockchain state -* The rdPoS protocol -* The HTTP and P2P servers +* The blockchain state (which contains the rdPoS protocol in itself) +* The HTTP server +* The consensus protocol The **Syncer** class is responsible for syncing the blockchain with other nodes in the network, as well as handling proper transaction broadcasts and block creations (if the node happens to be a Validator). @@ -27,13 +28,23 @@ The **Syncer** class is responsible for syncing the blockchain with other nodes The `consensus.h` file contains the **Consensus** class - responsible for processing blocks and transactions in the blockchain and applying the network's consensus rules into them. +## Database dumping + +The `dump.h` file contains the **Dumpable**, **DumpManager** and **DumpWorker** classes - together they are responsible for dumping the blockchain's components from memory to disk when required (e.g. blocks, transactions, contracts, the state itself, etc.) + +*Dumpable* is an abstraction of a dumpable object - that is, any component that inherits it is able to dump itself to disk. All classes that inherit it must implement their own dumping routine accordingly. + +*DumpManager* is the class that manages a list of Dumpable components in the blockchain, iterating through each of them and dumping them one by one when told to. + +*DumpWorker* acts as a worker thread for DumpManager, doing the actual work of dumping each Dumpable component sent by DumpManager in a separate thread. This allows for parallel dumps, speeding up the process significantly, which is important when there are many objects or particularly heavy ones (e.g. State) that need to be dumped ASAP without hanging the entire blockchain. + ## rdPoS -The `rdpos.h` file contains the **rdPoS** class - the implementation of the _Random Deterministic Proof of Stake_ algorithm used by the AppLayer network - as well as the **rdPoSWorker** and **Validator** classes. rdPoS is also considered a smart contract, but remains part of the AppLayer core protocol. +The `rdpos.h` file contains the **rdPoS** class - the implementation of the *Random Deterministic Proof of Stake* algorithm used by the AppLayer network - as well as the **Validator** class. rdPoS is also considered a smart contract (it derives from **BaseContract**), but as it is an essential part of the AppLayer core protocol, it remains in the `core` folder. ## State -The `state.h` file contains the **State** class - an abstraction of the blockchain's current state of accounts, balances, nonces, transactions, token balances, deployed contracts and emitted events at the current block in the network, responsible for owning and maintaining all of those and a few other shared inner variables. +The `state.h` file contains the **State** class - an abstraction of the blockchain's current state of accounts, balances, nonces, transactions, token balances and deployed contracts at the current block in the network, responsible for owning and maintaining all of those and a few other shared inner variables. A node's state and its data can only be altered through the process of block creation, either by creating a block itself, or receiving one from the network. In AppLayer's case, the class is often used for querying account data (current balance and nonce) and also processing and validating blocks and their transactions, as well as contract calls, following requirements such as (not limited to, but those are some of the most common): @@ -46,20 +57,8 @@ Not all functions from the class update the state. Check the [Doxygen](https://d ## Storage -The `storage.h` file contains the **Storage** class - an abstraction of the blockchain's history, maintaining a collection of blocks approved and validated by the network, other nodes, or itself through time. Those blocks store transactions, contracts, accounts, and can't be altered once they're in the blockchain, only searched for or read from. - -On node initialization, a history of up to 1000 of the most recent blocks is loaded into memory. Those blocks were stored in a previous initialization in the database. If there are no blocks (e.g. the blockchain was just deployed and initialized for the first time), a "genesis" block is automatically created and loaded in memory. +The `storage.h` file contains the **Storage** class - an abstraction of the blockchain's history, storing and maintaining a collection of blocks approved and validated by the network, other nodes, or itself through time. Those blocks store transactions, contracts, accounts, emitted contract events (if the node is initialized with the `RPC_TRACE` indexing mode set) and other blockchain-related data. Said data can't be altered once it's in the blockchain, it can only be searched for or read from. -Once a block and its transactions are received from the network, they're stored in memory. If more than 1000 blocks (or 1 million transactions) are stored in memory at a given time, older blocks are periodically saved to the database. This makes the blockchain lightweight memory-wise and extremely responsive. +On node initialization, if there are no blocks (e.g. the blockchain was just deployed and initialized for the first time), a "genesis" block is automatically created and loaded in memory. The "latest" block is always kept in memory, with subsequent older blocks being dumped to and queried constantly from the database. This makes the blockchain lightweight memory-wise and extremely responsive. Searching for and reading from blocks in history is done in several places in the system, so when the Storage and DB classes are working together, they act as the end point of the blockchain's operations history. - -## Database dumping - -The `dump.h` file contains the **Dumpable**, **DumpManager** and **DumpWorker** classes - together they are responsible for dumping the blockchain's components from memory to disk when required (e.g. blocks, transactions, contracts, the state itself, etc.) - -_Dumpable_ is an abstraction of a dumpable object - that is, any component that inherits it is able to dump itself to disk. All classes that inherit it must implement their own dumping routine accordingly. - -_DumpManager_ is the class that manages a list of Dumpable components in the blockchain, iterating through each of them and dumping them one by one when told to. - -_DumpWorker_ acts as a worker thread for DumpManager, doing the actual work of dumping each Dumpable component sent by DumpManager in a separate thread. This allows for parallel dumps, speeding up the process significantly, which is important when there are many objects or particularly heavy ones (e.g. State) that need to be dumped ASAP without hanging the entire blockchain. diff --git a/bdk-implementation/the-utils-folder.md b/bdk-implementation/the-utils-folder.md index caf943c..b347b78 100644 --- a/bdk-implementation/the-utils-folder.md +++ b/bdk-implementation/the-utils-folder.md @@ -8,11 +8,9 @@ This subchapter contains a brief overview of each one of the components inside t
-## FinalizedBlock - -The `finalizedblock.h` file contains the **FinalizedBlock** class - an abstraction of the structure of a block sent through the network and stored in the blockchain. A finalized block is inherently *final* - it cannot be modified anymore after construction. +## Clargs -The class only contains the bare structure and data of a block - it doesn't do any kind of operation, validation or verification on it. Having only finalized blocks across the entire project ensures block state integrity across all nodes. +The `clargs.h` file contains a few helper functions, structs and enums to parse command-line arguments passed to the project's executables. For an executable to be aware of the argument parser, it must be registered inside the **BDKTool** enum and the executable itself must call the `parseCommandLineArgs()` function, passing the arguments in a C-style manner (argc and argv) and the respective enum value. Check the executables' source files for more info. ## ContractReflectionInterface @@ -26,7 +24,7 @@ The `db.h` file contains the **DB** class - an abstraction of a [Speedb](https:/ The `dynamicexception.h` file contains the **DynamicException** class - a custom exception class inherited from `std::exception` used across the whole project. It is meant to be used when there's no applicable exception from the STD library for a given error that should be caught - usually the STD library is too generic, as the project grows some exceptions may become specific to the point we need to handle them in a customized manner. -## Secp256k1 +## ECDSA (Secp256k1) The `ecdsa.h` file contains the **Secp256k1** namespace - helper functions that abstract the functionalities of Bitcoin's [secp256k1](https://en.bitcoin.it/wiki/Secp256k1) elliptic curve cryptography library, used for handling, deriving and recovering private/public keys and addresses, as well as signing and verifying signed messages. @@ -36,9 +34,15 @@ The file also contains a few aliases for easier key handling, which are based on * **PubKey** (same as **FixedBytes<33>**) - alias for a _compressed_ public key * **UPubKey** (same as **FixedBytes<65>**) - alias for an _uncompressed_ public key +## FinalizedBlock + +The `finalizedblock.h` file contains the **FinalizedBlock** class - an abstraction of the structure of a block sent through the network and stored in the blockchain. A finalized block is inherently *final* - it cannot be modified anymore after construction. + +The class only contains the bare structure and data of a block - it doesn't do any kind of operation, validation or verification on it. Having only finalized blocks across the entire project ensures block state integrity across all nodes. + ## Hex -The `hex.h` file contains the **Hex** class - an abstraction of a strictly hex-formatted string (meaning it only accepts the characters within the range of `0x[1-9][a-f][A-F]`), which can also be set to strict or not (whether the string REQUIRES the `0x` prefix or not to be considered valid). Also contains aliases for working with raw-byte strings, such as **Byte, Bytes, BytesArr and BytesArrView**. +The `hex.h` file contains the **Hex** class - an abstraction of a strictly hex-formatted string (meaning it only accepts the characters within the range of `0x[1-9][a-f][A-F]`), which can also be set to strict or not (whether the string REQUIRES the `0x` prefix or not to be considered valid). Also contains aliases for working with raw-byte strings, such as **Byte, Bytes and BytesArr**. ## JsonAbi @@ -46,9 +50,9 @@ The `jsonabi.h` file contains the **JsonAbi** namespace - utility functions for ## Logger -The `logger.h` file contains the **Logger** class - a singleton responsible for logging any kind of info - and helper components such as the **Log** namespace (a namespace with predefined string names for referencing other modules), the **LogInfo** class (encapsulated log data), and the **LogType** enum (for flagging the severity of log messages). +The `logger.h` file contains the **Logger** class - a singleton responsible for logging any kind of info - and helper components such as the **Log** namespace (a namespace with predefined string names for referencing other modules), the **LogInfo** class (encapsulated log data), and the **LogType** enum (for flagging the severity of log messages). The `Logger::logToFile()` and `Logger::logToDebug()` functions print the given details to the respective `log.txt` and `debug.txt` files inside the node's directory. -The `Logger::logToFile()` and `Logger::logToDebug()` functions print the given details to the respective `log.txt` and `debug.txt` files inside the node's directory. +The file also contains a plethora of macros to leverage the logging functions and their flags in a more "hands-on" approach, usable anywhere in the project (with a few exceptions depending on the context of where the macro is used in code). Check the Doxygen comments in the file for more info on how to use them. ## Merkle @@ -57,7 +61,7 @@ The `merkle.h` class contains the **Merkle** class - a custom implementation of * [https://medium.com/coinmonks/implementing-merkle-tree-and-patricia-tree-b8badd6d9591](https://medium.com/coinmonks/implementing-merkle-tree-and-patricia-tree-b8badd6d9591) * [https://lab.miguelmota.com/merkletreejs/example/](https://lab.miguelmota.com/merkletreejs/example/) -A "Merkle Tree" is a data structure in binary tree format (e.g. "heap sort"), where data is stored in the "leaves", and the "branches" are paths to reach their data. This structure is commonly used in the crypto space as a tool for _verification_: it hashes the previous layers in pairs to make new layers, bottom-up, until it reaches a single result which would be the "root" of the tree - this makes the root a unique fingerprint for the entire tree, so you only need to check the root hash to verify that both the tree and its leaves were not tampered with. +A "Merkle Tree" is a data structure in binary tree format (e.g. "heap sort"), where data is stored in the "leaves", and the "branches" are paths to reach their data. This structure is commonly used in the crypto space as a tool for _verification_: it hashes the previous layers in pairs to make new layers, bottom-up, until it reaches a single result which would be the "root" of the tree - this makes the root a unique fingerprint for the entire tree, so you only need to check the root hash to verify both the tree and its leaves were not tampered with. ## Options @@ -73,47 +77,57 @@ For `RandomGen` to be useful, it needs to be seeded with a truly random number. ## SafeHash -The `safehash.h` contains the **SafeHash** struct - a custom hashing implementation for use with `std::unordered_map`, replacing the one used by default in C++'s STD library, like this for example: `std::unordered_map cachedBlocks;`. +The `safehash.h` contains the **SafeHash** struct - a custom hashing implementation for use with `unordered_map` and/or derivatives, replacing the one used by default, like this for example: `std::unordered_map`. -Essentially, the STD implementation of `unordered_map` uses `uint64_t` hashes, which is vulnerable to a potentially dangerous edge case where collisions could happen by having an enormous number of accounts and distributing them in a way that they have the same hash across all nodes. +Previously, we used `std::unordered_map` in conjunction with [a custom fix from this CodeForces article](https://codeforces.com/blog/entry/62393) so we could continue using the C++ STD's implementation of `unordered_map` due to its blazing fast query times (basically the STD implementation uses `uint64_t` hashes, which is vulnerable to a potentially dangerous edge case where collisions could happen by having an enormous number of accounts and distributing them in a way that they have the same hash across all nodes - this fix is not perfect, since it still uses `uint64_t`, but it's better than nothing since nodes keep different hashes). -[This article from CodeForces](https://codeforces.com/blog/entry/62393) provides a fix, which we implemented so we could continue using `std::unordered_map` normally, as it has blazing fast query times. It's not a perfect fix, since it still uses `uint64_t`, but it's better than nothing since nodes keep different hashes. +Currently, we replaced the usage of `std::unordered_map` in the whole project with Boost's implementation (`boost::unordered_flat_map`) in conjunction with the [Wyhash](https://github.com/wangyi-fudan/wyhash) library, the highest and fastest quality hash function available for size_t (64-bit) hashes, to achieve a simpler, faster and more solid functionality. The file also contains the **FNVHash** struct - a custom implementation of the [Fowler-Noll-Vo](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo\_hash\_function) hash struct, used within broadcast messages. This skips compiler optimizations and forces the whole string to be hashed, diminishing the chance of collisions (because, again, we're using 64-bit hashes). ## FixedBytes and its child classes -The `strings.h` file contains the **FixedBytes** template class - an abstraction of a normal `std::array` with a fixed size. For example, a `FixedBytes<10>` would have _exactly_ 10 characters in it - no more, no less. If initialized as an empty string, it will remain as a 10-character string nonetheless, with all characters set to "empty" (or `\x00` to be more exact). +The `strings.h` file contains the **FixedBytes** template class - an abstraction of a normal `std::array` with a fixed size. For example, a `FixedBytes<10>` would have *exactly* 10 characters in it - no more, no less. If initialized as an empty string, it will remain as a 10-character string nonetheless, with all characters set to "empty" (or `\x00` to be more exact). -Even though FixedBytes can be used on its own (_it's meant to store only bytes_, after all), it also serves as a base for specific classes, also declared within the same file and created with the intent of dealing with the many different ways that data strings are managed and transferred through the project in a better, less confusing and less convoluted way. They all inherit from the base FixedBytes class, applying fixed sizes of their own: +Even though FixedBytes can be used on its own (*it's meant to store only bytes*, after all), it also serves as a base for specific classes, also declared within the same file and created with the intent of dealing with the many different ways that data strings are managed and transferred through the project in a better, less confusing and less convoluted way: * **Hash** inherits **FixedBytes<32>** and abstracts a given 32-byte hash -* **Functor** inherits **FixedBytes<4>** and abstracts the first 4 bytes of a Solidity function's keccak hash +* **Functor** abstracts the first 4 bytes of a Solidity function's keccak hash, but does not inherit **FixedBytes** directly - instead it opts for a more practical approach and just treats those bytes as a `uint32_t` * **Signature** inherits **FixedBytes<65>** and abstracts a given full ECDSA signature (r, s and v) * **Address** inherits **FixedBytes<20>** and abstracts a given 20-byte address -* **StorageKey** inherits **FixedBytes<52>** and abstracts an EVM storage key (address + slot key) +* **StorageKey** inherits **FixedBytes<52>** and abstracts an EVM storage key (20 bytes address + 32 bytes slot key) + +All of these custom types are standard compliant (trivially copyable and trivially destructible), which means they can be handled like a STD container such as `std::vector` for example. ## TxBlock and TxValidator -The `tx.h` file contains the **TxBlock** and **TxValidator** classes - abstractions for a block transaction and a Validator transaction, respectively. The implementation logic and details for transactions within AppLayer are derived from the "Account" model, used by Ethereum and implemented by the [Aleth](https://github.com/ethereum/aleth) library, which is different from the "UTXO" model used by Bitcoin. +The `tx.h` file contains the **TxBlock** and **TxValidator** classes - abstractions for a block transaction and a Validator transaction, respectively. The implementation logic and details for those transactions are derived from the "Account" model, used by Ethereum and implemented by the [Aleth](https://github.com/ethereum/aleth) library, which is different from the "UTXO" model used by Bitcoin. + +It also contains a helper struct called `TxAdditionalData`, which contains metadata about a contract that was deployed in the chain, such as the transaction hash, how much gas was used in the transaction, if the call succeeded or not, and the contract's address. + +## UintConv, IntConv, StrConv and EVMCConv + +The respective files `uintconv.h`, `intconv.h`, `strconv.h` and `evmcconv.h` contain several namespaces related to aliases, conversion and manipulation of specific types of data (previously in the **Utils** namespace, now divided into their own namespaces): + +* **UintConv**: contains several `uintX_t` primitive type aliases used across the project, as well as their respective conversion functions (e.g. `uintXToBytes()`/`bytesToUintX()`) +* **IntConv**: contains several `intX_t` primitive type aliases used across the project, as well as their respective conversion functions (e.g. `intXToBytes()`/`bytesToIntX()`) +* **StrConv**: contains a few functions for converting and manipulating raw byte and UTF-8 strings (e.g. `padLeft()`/`padRight()` and their respective raw byte counterparts, `toLower()`/`toUpper()`, `bytesToString()`, `stringToBytes()`, etc.) +* **EVMCConv**: contains a few functions for converting and manipulating EVMC-specific data types (e.g. functors and their specific implementation of uint256) ## Utils -The `utils.h` file contains the **Utils** namespace - a place for more generalized miscellaneous utility functions, namespaces, enums and typedefs used throughout the BDK. +The `utils.h` file contains the **Utils** namespace - a place for generalized miscellaneous utility functions, namespaces, enums and typedefs used across the BDK. This list is only an example and does not reflect the entire contents of the file. We suggest you read the [Doxygen](https://doxygen.nl/) docs for more info about the class: -* Aliases for working with: - * Raw-byte strings (`Byte`,`Bytes`, `BytesArr`, `BytesArrView`), as well as helper functions for converting and/or manipulating them (e.g. `appendBytes()`) - * Unsigned integer types (`uintX_t`, `SafeUintX_t`) -* A map with addresses for Protocol Contracts (e.g. `rdPoS` and `ContractManager`) +* Helper functions that deal with printing (`safePrint()`, `safePrintTest()`, `printXYZ()`, etc.) +* Aliases for working with raw-byte strings (`Byte`,`Bytes`, `BytesArr`) and helper functions for converting and/or manipulating them (e.g. `appendBytes()`) + * For `appendBytes()` specifically, it is recommended to use it if you need a buffer, otherwise you can use `bytes::join()` as a slightly faster replacement (e.g. if you have all the data required at once, use `bytes::join()`, if you have the data scattered across different places, use a `Bytes` object as a buffer and use it with `appendBytes()`) +* The `ProtocolContractAddress` map for storing addresses for deployed Protocol Contracts (e.g. `rdPoS` and `ContractManager`) * Enums for network types (`Networks`), contract function types (`FunctionTypes`) and contract types (`ContractType`) * The `Account` struct, used to maintain account balance and nonce statuses, as well as contract-specific data (if the account represents a contract) * A wrapper for a pointer that ensures the pointer is never null (`NonNullUniquePtr`), as well as a wrapper for a pointer that forcefully nullifies it on destruction (`PointerNullifier`) * The `EventParam` struct, used for abstracting a contract event parameter -* `safePrint()`, used to print details to the node's terminal (which can be checked by attaching a tmux session to it) +* Several templated helper functions that deal with tuples, as well as helper functions that deal with Functors * The `sha3()` function, used extensively as the primary hash function for the entire project -* Functions that convert byte strings to unsigned integers and vice-versa (e.g. `uint256ToBytes()` and `bytesToUint256()`), as well as raw byte strings to normal string and vice-versa (e.g. `bytesToString()` and `stringToBytes()`) -* `padLeft()` and `padRight()`, used for adding padding to strings at their left and right sides, respectively -* `padLeftBytes()` and `padRightBytes()`, same as above, but specifically for use with raw byte strings -* `toLower()` and `toUpper()`, used for converting strings to all-lower and all-upper case, respectively +* The `randBytes()` function, used extensively as a random bytes string generator diff --git a/bdk-implementation/transactions-and-blocks.md b/bdk-implementation/transactions-and-blocks.md index 424c12e..2c21558 100644 --- a/bdk-implementation/transactions-and-blocks.md +++ b/bdk-implementation/transactions-and-blocks.md @@ -31,33 +31,35 @@ Depending on the type, a transaction will contain the following data fields once * **(TxBlock) value** - transaction value in its lowest unit * e.g. "satoshi" for BTC, "Wei" for ETH, etc. - "100000000" satoshis would be 1 BTC, "5000000000" Wei would be 0.000000005 ETH (or 5 gwei), so on and so forth * Since we're using an Ethereum-based format, we commonly refer to its terminology, so "value" is in "Wei" -* **(TxBlock) gasLimit** - maximum limit of gas units that the transaction will use, in Wei (e.g. "21000") - * If the transaction uses more than this limit, it will automatically fail - the original transaction value won't be spent, but what was already spent as gas is lost * **(TxBlock) maxPriorityFeePerGas** - value paid as an incentive for miners to include the transaction in the block, as per [EIP-1559](https://eips.ethereum.org/EIPS/eip-1559) * This is implemented but actively ignored since we don't have "miners", therefore only maxFeePerGas is counted * **(TxBlock) maxFeePerGas** - value paid by every unit of gas, in Gwei (e.g. "15" = 15000000000 Wei) * The total transaction fee is calculated as (gasLimit \* maxFeePerGas) - e.g. 21000 \* 15000000000 = 0.000315 ETH +* **(TxBlock) gasLimit** - maximum limit of gas units that the transaction will use, in Wei (e.g. "21000") + * If the transaction uses more than this limit, it will automatically fail - the original transaction value won't be spent, but what was already spent as gas is lost * **(Both)** ECDSA signature (Elliptic Curve Digital Signature Algorithm) for validating transaction integrity, split in three: * **r** - first half of the ECDSA signature (32 hex bytes) * **s** - second half of the ECDSA signature (32 hex bytes) * **v** - recovery ID (1 hex byte) +* **(Both) hash** - The transaction's own hash (including the signature), stored as a cache for performance reasons ## Block structure -Depending on the type, a block will contain the following conceptual structure: +A block will contain the following conceptual structure: -* **(FinalizedBlock only)** Validator signature (the one responsible for creating and signing the block) -* **(FinalizedBlock only)** Validator public key (for verifying the signature) +* Validator signature (the one responsible for creating and signing the block) +* Validator public key (for verifying the signature) * The block's header, which is made of: - * **(Both)** Previous block hash (a hash of the previous block's header, signed by the Validator) - * **(Both)** "Randomness" (a random seed generated by RandomGen during the previous block's creation) - * **(FinalizedBlock only)** Validator Tx Merkle Tree (to verify the integrity of Validator transactions) - * **(FinalizedBlock only)** Block Tx Merkle Tree (to verify the integrity of Block transactions) - * **(Both)** UNIX timestamp of the block, in microseconds - * **(Both)** Block height (commonly known as `nHeight`) -* **(Both)** List of block transactions -* **(Both)** List of Validator transactions -* **(Both)** A cached hash of the block's header + * Previous block hash (a hash of the previous block's header, signed by the Validator) + * "Randomness" (a random seed generated by RandomGen during the previous block's creation) + * Validator Tx Merkle Tree (to verify the integrity of Validator transactions) + * Block Tx Merkle Tree (to verify the integrity of Block transactions) + * UNIX timestamp of the block, in microseconds + * Block height (commonly known as `nHeight`) +* List of Validator transactions +* List of block transactions +* A cached hash of the block's header +* The total size of the block, in bytes In practice, a block is simply a serialized bytes string, transmitted through the network and stored in the blockchain, so it's up to the code logic to properly parse it. Another way to see the block in a more "raw" format would be ("bytes" are in hex format, e.g. `0xFF` = 1 byte): diff --git a/evm-contracts/README.md b/evm-contracts/README.md index 1fb5a78..c85b812 100644 --- a/evm-contracts/README.md +++ b/evm-contracts/README.md @@ -4,7 +4,6 @@ description: A primer on EVM smart contracts in the AppLayer ecosystem. # EVM contracts -Aside from native contracts, AppLayer can also execute Solidity contracts as-is by the use of the AppLayer EVM, which is compatible with bytecode deployment. This means any language that compiles to EVM bytecode (e.g. Solidity, Viper, etc.) can be used to deploy contracts in the AppLayer EVM in a seamless, straight-forward way. - -This kind of compatibility is possible thanks to the integration of the [EVMOne](https://github.com/ethereum/evmone) virtual machine (originally made by the Ethereum devs) and EVMC libraries. +Aside from native contracts, AppLayer can also execute Solidity contracts as-is by the use of the AppLayer EVM, which is compatible with bytecode deployment. This means any language that compiles to EVM bytecode (e.g. [Solidity](https://soliditylang.org/), [Vyper](https://docs.vyperlang.org/en/stable/), etc.) can be used to deploy contracts in the AppLayer EVM in a seamless, straight-forward way. +This kind of compatibility is possible thanks to the integration of the [EVMOne](https://github.com/ethereum/evmone) virtual machine (originally made by the Ethereum devs) and [EVMC](https://github.com/ethereum/evmc) libraries. diff --git a/evm-contracts/cpp-to-other-contract-calls.md b/evm-contracts/cpp-to-other-contract-calls.md index 77a4812..3fd9dfd 100644 --- a/evm-contracts/cpp-to-other-contract-calls.md +++ b/evm-contracts/cpp-to-other-contract-calls.md @@ -7,49 +7,18 @@ description: How contract calls happen from the C++ side in AppLayer. The `ContractHost` class employs templated functions to support flexible and efficient interaction with contracts. These templates enable passing any combination of arguments and return types (including `void`) to and from other types of contracts. This use of templates helps to leverage the fast ABI encoding/decoding processes, ensuring optimal performance and flexibility during contract execution: ```cpp -template R -callContractViewFunction( - const BaseContract* caller, - const Address& targetAddr, - R(C::*func)(const Args&...) const, const - Args&... args) const; - -template R -callContractViewFunction( - const BaseContract* caller, - const Address& targetAddr, - R(C::*func)() const) const; - -template -requires (!std::is_same::value) -R callContractFunction( - BaseContract* caller, const Address& targetAddr, - const uint256_t& value, - R(C::*func)(const Args&...), const Args&... args -) - template -requires (std::is_same::value) -void callContractFunction( - BaseContract* caller, const Address& targetAddr, - const uint256_t& value, - R(C::*func)(const Args&...), const Args&... args -) +R callContractViewFunction( + const BaseContract* caller, const Address& targetAddr, + R(C::*func)(const Args&...) const, const Args&... args +) const; template -requires (!std::is_same::value) R callContractFunction( BaseContract* caller, const Address& targetAddr, const uint256_t& value, R(C::*func)(const Args&...), const Args&... args -) - -template -requires (!std::is_same::value) -R callContractFunction( - BaseContract* caller, const Address& targetAddr, - const uint256_t& value, R(C::*func)() -) +); ``` This approach allows for dynamic interaction with contracts without pre-defining all possible function signatures, accommodating various contract behaviors and states dynamically. diff --git a/evm-contracts/evm-to-other-contract-calls.md b/evm-contracts/evm-to-other-contract-calls.md index 363d093..a5d0253 100644 --- a/evm-contracts/evm-to-other-contract-calls.md +++ b/evm-contracts/evm-to-other-contract-calls.md @@ -14,33 +14,40 @@ This function is designed to handle both C++ and EVM contract calls, as shown be ```cpp evmc::Result ContractHost::call(const evmc_message& msg) noexcept { - Address recipient(msg.recipient); - auto &recipientAccount = *accounts_[recipient]; // We need to take a reference to the account, not a reference to the pointer. - this->leftoverGas_ = msg.gas; - /// evmc::Result constructor is: _status_code + _gas_left + _output_data + _output_size - if (recipientAccount.contractType == CPP) { - // We are a CPP contract, we need to call the contract evmEthCall function and put the result into a evmc::Result - try { - this->deduceGas(1000); // CPP contract call is 1000 gas - auto& contract = contracts_[recipient]; - if (contract == nullptr) { - throw DynamicException("ContractHost call: contract not found"); - } - this->setContractVars(contract.get(), Address(msg.sender), Utils::evmcUint256ToUint256(msg.value)); - Bytes ret = contract->evmEthCall(msg, this); - return evmc::Result(EVMC_SUCCESS, this->leftoverGas_, 0, ret.data(), ret.size()); - } catch (std::exception& e) { - this->evmcThrows_.emplace_back(e.what()); - this->evmcThrow_ = true; - return evmc::Result(EVMC_PRECOMPILE_FAILURE, this->leftoverGas_, 0, nullptr, 0); - } + evmc::Result result; + const bool isContractCall = isCall(msg); + + if (isContractCall) { + this->traceCallStarted(msg); + } + + switch (this->decodeContractCallType(msg)) + { + case ContractType::CREATE: { + result = this->callEVMCreate(msg); + break; + } + case ContractType::CREATE2: { + result = this->callEVMCreate2(msg); + break; + } + case ContractType::PRECOMPILED: { + result = this->processBDKPrecompile(msg); + break; } - evmc::Result result (evmc_execute(this->vm_, &this->get_interface(), this->to_context(), - evmc_revision::EVMC_LATEST_STABLE_REVISION, &msg, - recipientAccount.code.data(), recipientAccount.code.size())); - this->leftoverGas_ = result.gas_left; // gas_left is not linked with leftoverGas_, we need to link it. - this->deduceGas(5000); // EVM contract call is 5000 gas - result.gas_left = this->leftoverGas_; // We need to set the gas left to the leftoverGas_ + case ContractType::CPP: { + result = this->callCPPContract(msg); + break; + } + default: + result = this->callEVMContract(msg); + break; + } + + if (isContractCall) { + this->traceCallFinished(result.raw()); + } + return result; } ``` diff --git a/evm-contracts/seamless-cpp-evm-integration.md b/evm-contracts/seamless-cpp-evm-integration.md index e427bfb..1fa6b9d 100644 --- a/evm-contracts/seamless-cpp-evm-integration.md +++ b/evm-contracts/seamless-cpp-evm-integration.md @@ -30,18 +30,18 @@ struct evmc_message { `ContractHost` plays a critical role in distinguishing whether a contract is implemented in C++ or EVM and executing it accordingly. Below is an example illustrating how C++ contracts can invoke functions in other contracts, whether they are C++ or EVM: -
template <typename R, typename C, typename... Args>
-requires (!std::is_same<R, void>::value)
-R callContractFunction(
-  BaseContract* caller, const Address& targetAddr,
-  const uint256_t& value,
-  R(C::*func)(const Args&...), const Args&... args
+```c++
+template 
+R callContractFunctionImpl(
+  BaseContract* caller, const Address& targetAddr,
+  const uint256_t& value,
+  R(C::*func)(const Args&...), const Args&... args
 ) {
   // 1000 Gas Limit for every C++ contract call!
-  auto& recipientAcc = *this->accounts_[targetAddr];
+  auto& recipientAcc = *this->accounts_[targetAddr];
   if (!recipientAcc.isContract()) {
     throw DynamicException(std::string(__func__) + ": Contract does not exist - Type: "
-      + Utils::getRealTypeName<C>() + " at address: " + targetAddr.hex().get()
+      + Utils::getRealTypeName() + " at address: " + targetAddr.hex().get()
     );
   }
   if (value) {
@@ -49,7 +49,7 @@ R callContractFunction(
   }
   NestedCallSafeGuard guard(caller, caller->caller_, caller->value_);
   switch (recipientAcc.contractType) {
-    case ContractType::EVM : {
+    case ContractType::EVM: {
       this->deduceGas(10000);
       evmc_message msg;
       msg.kind = EVMC_CALL;
@@ -62,35 +62,41 @@ R callContractFunction(
       if (functionName.empty()) {
         throw DynamicException("ContractHost::callContractFunction: EVM contract function name is empty (contract not registered?)");
       }
-      auto functor = ABI::FunctorEncoder::encode<Args...>(functionName);
-      Bytes fullData;
-      Utils::appendBytes(fullData, Utils::uint32ToBytes(functor.value));
-      Utils::appendBytes(fullData, ABI::Encoder::encodeData<Args...>(args...));
+      auto functor = ABI::FunctorEncoder::encode(functionName);
+      Bytes fullData;
+      Utils::appendBytes(fullData, UintConv::uint32ToBytes(functor.value));
+      if constexpr (sizeof...(Args) > 0) {
+        Utils::appendBytes(fullData, ABI::Encoder::encodeData(args...));
+      }
       msg.input_data = fullData.data();
       msg.input_size = fullData.size();
-      msg.value = Utils::uint256ToEvmcUint256(value);
+      msg.value = EVMCConv::uint256ToEvmcUint256(value);
       msg.create2_salt = {};
       msg.code_address = targetAddr.toEvmcAddress();
-      evmc::Result result (evmc_execute(this->vm_, &this->get_interface(), this->to_context(),
-      evmc_revision::EVMC_LATEST_STABLE_REVISION, &msg, recipientAcc.code.data(), recipientAcc.code.size()));
+      evmc::Result result (evmc_execute(this->vm_, &this->get_interface(), this->to_context(),
+      evmc_revision::EVMC_LATEST_STABLE_REVISION, &msg, recipientAcc.code.data(), recipientAcc.code.size()));
       this->leftoverGas_ = result.gas_left;
       if (result.status_code) {
-        auto hexResult = Hex::fromBytes(BytesArrView(result.output_data, result.output_data + result.output_size));
+        auto hexResult = Hex::fromBytes(bytes::View(result.output_data, result.output_data + result.output_size));
         throw DynamicException("ContractHost::callContractFunction: EVMC call failed - Type: "
-          + Utils::getRealTypeName<C>() + " at address: " + targetAddr.hex().get() + " - Result: " + hexResult.get()
+          + Utils::getRealTypeName() + " at address: " + targetAddr.hex().get() + " - Result: " + hexResult.get()
         );
       }
-      return std::get<0>(ABI::Decoder::decodeData<R>(BytesArrView(result.output_data, result.output_data + result.output_size)));
+      if constexpr (std::same_as) {
+        return;
+      } else {
+        return std::get<0>(ABI::Decoder::decodeData(bytes::View(result.output_data, result.output_data + result.output_size)));
+      }
     } break;
-    case ContractType::CPP : {
+    case ContractType::CPP: {
       this->deduceGas(1000);
-      C* contract = this->getContract<C>(targetAddr);
+      C* contract = this->getContract(targetAddr);
       this->setContractVars(contract, caller->getContractAddress(), value);
       try {
         return contract->callContractFunction(this, func, args...);
-      } catch (const std::exception& e) {
+      } catch (const std::exception& e) {
         throw DynamicException(e.what() + std::string(" - Type: ")
-          + Utils::getRealTypeName<C>() + " at address: " + targetAddr.hex().get()
+          + Utils::getRealTypeName() + " at address: " + targetAddr.hex().get()
         );
       }
     }
@@ -99,4 +105,4 @@ R callContractFunction(
     }
   }
 }
-
+``` diff --git a/evm-contracts/state-management-and-vm-instance-creation.md b/evm-contracts/state-management-and-vm-instance-creation.md index 5e3f053..da89022 100644 --- a/evm-contracts/state-management-and-vm-instance-creation.md +++ b/evm-contracts/state-management-and-vm-instance-creation.md @@ -9,11 +9,13 @@ The VM itself is owned and instantiated by the `State` class, which reflects a c ```cpp ContractHost( evmc_vm* vm, - EventManager& eventManager, + DumpManager& manager, const Storage& storage, + const Hash& randomnessSeed, const evmc_tx_context& currentTxContext, - std::unordered_map, SafeHash>& accounts, - std::unordered_map& vmStorage, + boost::unordered_flat_map, SafeHash>& contracts, + boost::unordered_flat_map, SafeHash>& accounts, + boost::unordered_flat_map& vmStorage, const Hash& txHash, const uint64_t txIndex, const Hash& blockHash, diff --git a/introducing-applayer/what-is-applayer.md b/introducing-applayer/what-is-applayer.md index 1d16748..d298be3 100644 --- a/introducing-applayer/what-is-applayer.md +++ b/introducing-applayer/what-is-applayer.md @@ -8,7 +8,7 @@ The AppLayer Network is made up of three parts: * A Blockchain Development Kit (hereby denominated [**BDK**](https://github.com/AppLayerLabs/bdk-cpp)), with extensive documentation for developers to easily build their own AppLayer's with unprecedented freedom * An EVM network built on top of the Blockchain Development Kit which enables builders to deploy EVM smart contracts and scale with C++ stateful pre-compiles -* A network that allows the bridging of data and assets between these app-specific chains and external chains we call the Chain Abstraction Network (**CAN**) +* A network that allows the bridging of data and assets between these app-specific chains and external chains - we call it the Chain Abstraction Network (**CAN**) Therefore, blockchains built using the BDK are able to communicate with each other through AppLayer. diff --git a/join-our-community.md b/join-our-community.md index 7e4b53d..bd32a10 100644 --- a/join-our-community.md +++ b/join-our-community.md @@ -12,12 +12,12 @@ AppLayer is empowering the next generation of **#Web3** builders! We share insig **Discord:** [https://discord.gg/mSheqXQrQm](https://discord.gg/mSheqXQrQm) -**Twitter:** [https://twitter.com/AppLayerLabs](https://twitter.com/SparqNet) +**Twitter:** [https://twitter.com/AppLayerLabs](https://twitter.com/AppLayerLabs) **Telegram:** [https://t.me/AppLayerLabs](https://t.me/AppLayerLabs) **Telegram Announcements:** [https://t.me/AppLayer\_News](https://t.me/AppLayer\_News) -**Medium:** [https://medium.com/@AppLayerLabs](https://medium.com/@SparqNet) +**Medium:** [https://medium.com/@AppLayerLabs](https://medium.com/@AppLayerLabs) -**LinkedIn:** [https://www.linkedin.com/company/sparq-network](https://www.linkedin.com/company/sparq-network) +**LinkedIn:** [https://www.linkedin.com/company/applayerlabs](https://www.linkedin.com/company/applayerlabs) diff --git a/precompiled-contracts/README.md b/precompiled-contracts/README.md index 5b5f8cd..8da55d9 100644 --- a/precompiled-contracts/README.md +++ b/precompiled-contracts/README.md @@ -4,8 +4,8 @@ description: A primer on natively-coded smart contracts in the AppLayer ecosyste # Precompiled contracts -Precompiled contracts (also known as "native contracts" or "stateful pre-compiles" - "stateful" since they can maintain a state, and "pre-compile" because they're compiled machine code not interpreted by a virtual machine) are contracts coded with the blockchain's native language, with things like transaction parsing methods and arguments, as well as management and storage of the contract's variables in a database, being manually coded in e.g. C++ to be tightly integrated with the blockchain itself. +Precompiled contracts (also known as "native contracts" or "stateful pre-compiles" - "stateful" since they can maintain a state, and "pre-compile" because they're compiled machine code not interpreted by a virtual machine) are contracts coded with the blockchain's native language, with things like transaction parsing methods and arguments, as well as management and storage of the contract's variables in a database, being manually coded in the blockchain's native language (e.g. bdk-cpp in C++) to be tightly integrated with the blockchain itself. Similar to Solidity contracts, they can be used to employ any type of logic within the network, but unlike Solidity, they aren’t subject to EVM constraints. This means we can take advantage of that fact and have full control of the contract's logic, unleashing blazing fast performance, flexibility and power. -The contract templates provided by AppLayer's BDK (in the `src/contract/templates` folder) are based on OpenZeppelin contracts, maintaining the same operational standards known in the Solidity ecosystem, but on native C++. +The contract templates provided by AppLayer's BDK (in the `src/contract/templates` folder) are based on OpenZeppelin contracts, maintaining the same operational standards known in the Solidity ecosystem, but on native C++ (in the case of bdk-cpp). diff --git a/precompiled-contracts/creating-a-dynamic-contract-advanced.md b/precompiled-contracts/creating-a-dynamic-contract-advanced.md index 8812e5f..de0cdc5 100644 --- a/precompiled-contracts/creating-a-dynamic-contract-advanced.md +++ b/precompiled-contracts/creating-a-dynamic-contract-advanced.md @@ -81,6 +81,7 @@ Inside `erc20wrapper.h`, let's implement the header (comments were taken out so #include #include "../../utils/db.h" +#include "../../utils/utils.h" #include "../abi.h" #include "../contractmanager.h" #include "../dynamiccontract.h" @@ -89,26 +90,19 @@ Inside `erc20wrapper.h`, let's implement the header (comments were taken out so class ERC20Wrapper : public DynamicContract { private: - SafeUnorderedMap> tokensAndBalances_; + SafeUnorderedMap> tokensAndBalances_; void registerContractFunctions() override; public: using ConstructorArguments = std::tuple<>; - ERC20Wrapper( - const Address& contractAddress, const DB& db - ); - - ERC20Wrapper( - const Address& address, const Address& creator, - const uint64_t& chainId - ); + ERC20Wrapper(const Address& contractAddress, const DB& db); + ERC20Wrapper(const Address& address, const Address& creator, const uint64_t& chainId); + ~ERC20Wrapper() override; static void registerContract() { ContractReflectionInterface::registerContractMethods< - ERC20Wrapper, - const Address&, const Address&, const uint64_t&, - DB& + ERC20Wrapper, const Address&, const Address&, const uint64_t&, DB& >( std::vector{}, std::make_tuple("getContractBalance", &ERC20Wrapper::getContractBalance, FunctionTypes::View, std::vector{"token"}), @@ -128,7 +122,7 @@ class ERC20Wrapper : public DynamicContract { DBBatch dump() const override; }; -#endif // ERC20WRAPPER_H +#endif ``` Here, we recreated the contract's functions but also added a few extra functions (explained in the previous sections). In short, we create: @@ -148,13 +142,11 @@ Inside `erc20wrapper.cpp`, let's implement both constructors and the dumping fun ```cpp #include "erc20wrapper.h" -EERC20Wrapper::ERC20Wrapper( - ContractManagerInterface& interface, const Address& contractAddress, DB& db -) : DynamicContract(interface, contractAddress, db), tokensAndBalances_(this) +ERC20Wrapper::ERC20Wrapper(const Address& contractAddress, const DB& db +) : DynamicContract(contractAddress, db), tokensAndBalances_(this) { - auto tokensAndBalances = this->db_.getBatch(this->getNewPrefix("tokensAndBalances_")); - for (const auto& dbEntry : tokensAndBalances) { - BytesArrView valueView(dbEntry.value); + for (const auto& dbEntry : db.getBatch(this->getNewPrefix("tokensAndBalances_"))) { + bytes::View valueView(dbEntry.value); this->tokensAndBalances_[Address(dbEntry.key)][Address(valueView.subspan(0, 20))] = Utils::fromBigEndian(valueView.subspan(20)); } @@ -165,9 +157,8 @@ EERC20Wrapper::ERC20Wrapper( this->tokensAndBalances_.enableRegister(); } -ERC20Wrapper::ERC20Wrapper( - ContractManagerInterface& interface, const Address& address, const Address& creator, const uint64_t& chainId, DB& db -) : DynamicContract(interface, "ERC20Wrapper", address, creator, chainId, db), tokensAndBalances_(this) +ERC20Wrapper::ERC20Wrapper(const Address& address, const Address& creator, const uint64_t& chainId +) : DynamicContract("ERC20Wrapper", address, creator, chainId), tokensAndBalances_(this) { this->tokensAndBalances_.commit(); @@ -176,11 +167,13 @@ ERC20Wrapper::ERC20Wrapper( this->tokensAndBalances_.enableRegister(); } +ERC20Wrapper::~ERC20Wrapper() {} + DBBatch ERC20Wrapper::dump() const { DBBatch dbBatch = BaseContract::dump(); for (auto i = tokensAndBalances_.cbegin(); i != tokensAndBalances_.cend(); ++i) { for (auto j = i->second.cbegin(); j != i->second.cend(); ++j) { - const auto& key = i->first.get(); + const auto& key = i->first; Bytes value = j->first.asBytes(); Utils::appendBytes(value, Utils::uintToBytes(j->second)); dbBatch.push_back(key, value, this->getNewPrefix("tokensAndBalances_")); @@ -205,32 +198,28 @@ uint256_t ERC20Wrapper::getContractBalance(const Address& token) const { uint256_t ERC20Wrapper::getUserBalance(const Address& token, const Address& user) const { auto it = this->tokensAndBalances_.find(token); - if (it == this->tokensAndBalances_.end()) { - return 0; - } + if (it == this->tokensAndBalances_.cend()) return 0; auto itUser = it->second.find(user); - if (itUser == it->second.end()) { - return 0; - } + if (itUser == it->second.cend()) return 0; return itUser->second; } void ERC20Wrapper::withdraw(const Address& token, const uint256_t& value) { auto it = this->tokensAndBalances_.find(token); - if (it == this->tokensAndBalances_.end()) throw std::runtime_error("Token not found"); + if (it == this->tokensAndBalances_.end()) throw DynamicException("Token not found"); auto itUser = it->second.find(this->getCaller()); - if (itUser == it->second.end()) throw std::runtime_error("User not found"); - if (itUser->second <= value) throw std::runtime_error("ERC20Wrapper: Not enough balance"); + if (itUser == it->second.end()) throw DynamicException("User not found"); + if (itUser->second <= value) throw DynamicException("ERC20Wrapper: Not enough balance"); itUser->second -= value; this->callContractFunction(token, &ERC20::transfer, this->getCaller(), value); } void ERC20Wrapper::transferTo(const Address& token, const Address& to, const uint256_t& value) { auto it = this->tokensAndBalances_.find(token); - if (it == this->tokensAndBalances_.end()) throw std::runtime_error("Token not found"); + if (it == this->tokensAndBalances_.end()) throw DynamicException("Token not found"); auto itUser = it->second.find(this->getCaller()); - if (itUser == it->second.end()) throw std::runtime_error("User not found"); - if (itUser->second <= value) throw std::runtime_error("ERC20Wrapper: Not enough balance"); + if (itUser == it->second.end()) throw DynamicException("User not found"); + if (itUser->second <= value) throw DynamicException("ERC20Wrapper: Not enough balance"); itUser->second -= value; this->callContractFunction(token, &ERC20::transfer, to, value); } diff --git a/precompiled-contracts/creating-a-dynamic-contract-simple/README.md b/precompiled-contracts/creating-a-dynamic-contract-simple/README.md index 3072c6b..281a7de 100644 --- a/precompiled-contracts/creating-a-dynamic-contract-simple/README.md +++ b/precompiled-contracts/creating-a-dynamic-contract-simple/README.md @@ -4,7 +4,7 @@ description: A walkthrough on the basics of a Dynamic Contract's structure. # Creating a Dynamic Contract (Simple) -Let's create a test contract that allows three variables - `name`, `number` and `tuple` - to be changed by the owner of the contract, as well as perform basic operations with structs/tuples and event emissions. We will call this contract `SimpleContract` (which just so happens to be already implemented by the project due to internal testing purposes, but you can do it yourself by hand - check the `simplecontract.{h,cpp,sol}` files for reference). +Let's create a test contract that allows three variables - `name`, `number` and `tuple` - to be changed by the owner of the contract, as well as perform basic operations with structs/tuples and event emissions. We will call this contract `SimpleContract` (which just so happens to be already implemented by the project due to internal testing purposes, but you can do it yourself by hand - check the `src/contract/templates/simplecontract.{h,cpp,sol}` files for reference). ### Creating the Files diff --git a/precompiled-contracts/creating-a-dynamic-contract-simple/deploying-and-testing.md b/precompiled-contracts/creating-a-dynamic-contract-simple/deploying-and-testing.md index a791227..ed88a4d 100644 --- a/precompiled-contracts/creating-a-dynamic-contract-simple/deploying-and-testing.md +++ b/precompiled-contracts/creating-a-dynamic-contract-simple/deploying-and-testing.md @@ -56,17 +56,11 @@ Finally, write your tests. In this case, for `SimpleContract`, tests should look ```cpp #include "../../src/libs/catch2/catch_amalgamated.hpp" -#include "../../src/utils/db.h" -#include "../../src/utils/options.h" -#include "../../src/core/rdpos.h" -#include "../../src/contract/abi.h" -#include "../../src/contract/contractmanager.h" + #include "../../src/contract/templates/simplecontract.h" #include "../sdktestsuite.hpp" -#include - namespace TSimpleContract { TEST_CASE("SimpleContract class", "[contract][simplecontract]") { SECTION("SimpleContract creation") { @@ -129,6 +123,6 @@ namespace TSimpleContract { Keep in mind that we're not accessing the contract directly, we're interacting with it through `SDKTestSuite`, and that requires us to parse its inputs and outputs accordingly. -In order to run your tests, compile the project as you normally would, and then run `./src/bins/orbitersdkd-tests/orbitersdkd-tests -d yes [simplecontract]` from within your build directory. The `[simplecontract]` tag forces only these specific tests for SimpleContract to run (this is set in the `TEST_CASE()` lines in the example above). The `-d yes` flag makes it more verbose, showing exactly which test case is being run at the moment. +In order to run your tests, compile the project as you normally would, and then run `./src/bins/bdkd-tests/bdkd-tests -d yes [simplecontract]` from within your build directory. The `[simplecontract]` tag forces only these specific tests for SimpleContract to run (this is set in the `TEST_CASE()` lines in the example above). The `-d yes` flag makes it more verbose, showing exactly which test case is being run at the moment. As a bonus, even though this whole chapter is focused on precompiled contracts, the `SDKTestSuite` also has a function that can deploy a contract's bytecode directly, called `SDKTestSuite::deployBytecode()`, which takes a `Bytes` object as a parameter that represents the contract's bytecode. diff --git a/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-header.md b/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-header.md index dd39c3f..ce73b48 100644 --- a/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-header.md +++ b/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-header.md @@ -17,7 +17,7 @@ Open the header file (`simplecontract.h`) and add the following lines: #include "../dynamiccontract.h" #include "../variables/safestring.h" #include "../variables/safetuple.h" -#include "../../utils/utils.h" // SafeUintX_t aliases declared here +#include "../variables/safeuint.h" class SimpleContract : public DynamicContract { private: @@ -32,8 +32,7 @@ class SimpleContract : public DynamicContract { This is a simple skeleton so we can start building the proper contract. From top to bottom: * We create [include guards](https://en.wikipedia.org/wiki/Include\_guard) as a safety measure -* We include the `DynamicContract` class and three SafeVariable classes for the contract's inner variables - `SafeString`, `SafeUint256_t` and `SafeTuple`, which represent a Solidity `string`, `uint256` and `tuple/struct` types, respectively (check the `src/contract/variables` subfolder for all available variable abstractions) - * For SafeUint (and SafeInt as well), we include `src/utils/utils.h` as all of the SafeUintX/SafeIntX aliases are declared there, if you want to use `SafeUint_t`/`SafeInt_t` then include `src/contract/variables/safeuint.h` (or `safeint.h`) directly instead +* We include the `DynamicContract` class and three SafeVariable classes for the contract's inner variables - `SafeString`, `SafeUint256_t` and `SafeTuple`, which represent a Solidity `string`, `uint256` and `tuple/struct` types, respectively (check the `src/contract/variables` subfolder for all available variable abstractions) - for SafeUint (and SafeInt as well), you can either use `SafeUint_t`/`SafeInt_t` or the `SafeUintX_t`/`SafeIntX_t` aliases directly, as they are included in their respective header files * We create our `SimpleContract` class, inherit it from `DynamicContract`, and leave some space for the private and public members that will be coded next Now we can declare the members of our contract. We have to pay attention to some rules described earlier, which we'll go through slowly, one part at a time. @@ -107,7 +106,7 @@ class SimpleContract : public DynamicContract { // Constructor from load. Load contract from database. SimpleContract( const Address& address, - const std::unique_ptr &db + const DB& db ); // ... @@ -119,20 +118,19 @@ Aside from our `name`, `number` and `tuple` variables, both constructors also ta ```cpp // Constructor that creates the contract from scratch DynamicContract( - const std::string& contractName, const Address& address, const Address& creator, const uint64_t& chainId -) : BaseContract(contractName, address, creator, chainId) {}; +) : BaseContract("SimpleContract", address, creator, chainId) {}; // Constructor that loads the contract from the database DynamicContract( const Address& address, - const std::unique_ptr &db + const DB& db ) : BaseContract(address, db) {}; ``` -`address`, `creator`, `chainId` and `db` are internal variables used by the base class, and should _always_ be declared _last_. They are equivalent to: +`address`, `creator`, `chainId` and `db` are internal variables used by the base class, and should *always* be declared *last*. They are equivalent to: | DynamicContract constructor argument | Taken from | | ------------------------------------ | ------------------------------------------- | @@ -141,7 +139,7 @@ DynamicContract( | chainId | this->options->getChainID() | | DB | this->db | -Keep in mind that, when calling the base class constructor later on, the `contractName` argument MUST be EXACTLY the same as your contract's class name. This is because `contractName` is used to load the contract type from the database, so incorrectly naming it will result in a segfault at load time. +Keep in mind that, when calling the base class constructor later on, the contract's name argument (in this case, "SimpleContract") MUST be EXACTLY the same as your contract's class name. This is because the name is used to load the contract type from the database, so incorrectly naming it will result in a segfault at load time. ## Declaring the Contract Events @@ -181,7 +179,7 @@ class SimpleContract : public DynamicContract { * The first element is the argument type (e.g. `name` is a `std::string`, `number` is a `uint256_t`, `tuple` is a `std::tuple`) * The second element is a bool that indicates whether the argument should be indexed or not. * If your event has no arguments at all you can omit it or pass an empty tuple instead (e.g. `this->emitEvent(__func__, std::make_tuple())`) -* **(Optional)** A flag that indicates whether the event is anonymous or not. Events are non-anonymous by default, so if you wish you can omit this (which is the case for our example, it is equivalent to `this->emitEvent(__func__, std::make_tuple(name), false)` - if it was an anonymous event, the last flag would be `true` instead) +* **(Optional)** A flag that indicates whether the event is anonymous or not. Events are non-anonymous by default (the flag defaults to `false`), so if you wish you can omit this (which is the case for our example, it is equivalent to `this->emitEvent(__func__, std::make_tuple(name), false)` - if it was an anonymous event, the last flag would be `true` instead) ## Registering the Contract Class @@ -210,8 +208,7 @@ class SimpleContract : public DynamicContract { ContractReflectionInterface::registerContractMethods< SimpleContract, const std::string&, const uint256_t&, const std::tuple&, ContractManagerInterface&, - const Address&, const Address&, const uint64_t&, - const std::unique_ptr& + const Address&, const Address&, const uint64_t&, const DB& >( std::vector{"name_", "number_", "tuple_"}, std::make_tuple("getName", &SimpleContract::getName, FunctionTypes::View diff --git a/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-source.md b/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-source.md index 4c43f5a..4a479db 100644 --- a/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-source.md +++ b/precompiled-contracts/creating-a-dynamic-contract-simple/simple-contract-source.md @@ -24,6 +24,9 @@ Our source file will look something like this: ```cpp #include "simplecontract.h" +#include "../../utils/uintconv.h" +#include "../../utils/utils.h" + SimpleContract::SimpleContract( const std::string& name, const uint256_t& number, @@ -53,11 +56,11 @@ SimpleContract::SimpleContract( const Address& address, const DB& db ) : DynamicContract(address, db), name_(this), number_(this), tuple_(this) { - this->name_ = Utils::bytesToString(db.get(std::string("name_"), this->getDBPrefix())); - this->number_ = Utils::bytesToUint256(db.get(std::string("number_"), this->getDBPrefix())); + this->name_ = StrConv::bytesToString(db.get(std::string("name_"), this->getDBPrefix())); + this->number_ = UintConv::bytesToUint256(db.get(std::string("number_"), this->getDBPrefix())); this->tuple_ = std::make_tuple( - Utils::bytesToString(db.get(std::string("tuple_name"), this->getDBPrefix())), - Utils::bytesToUint256(db.get(std::string("tuple_number"), this->getDBPrefix())) + StrConv::bytesToString(db.get(std::string("tuple_name"), this->getDBPrefix())), + UintConv::bytesToUint256(db.get(std::string("tuple_number"), this->getDBPrefix())) ); this->name_.commit(); @@ -71,13 +74,12 @@ SimpleContract::SimpleContract( this->tuple_.enableRegister(); } - DBBatch SimpleContract::dump() const { DBBatch dbBatch; - dbBatch.push_back(Utils::stringToBytes("name_"), Utils::stringToBytes(this->name_.get()), this->getDBPrefix()); - dbBatch.push_back(Utils::stringToBytes("number_"), Utils::uint256ToBytes(this->number_.get()), this->getDBPrefix()); - dbBatch.push_back(Utils::stringToBytes("tuple_name"), Utils::stringToBytes(get<0>(this->tuple_)), this->getDBPrefix()); - dbBatch.push_back(Utils::stringToBytes("tuple_number"), Utils::uint256ToBytes(get<1>(this->tuple_)), this->getDBPrefix()); + dbBatch.push_back(StrConv::stringToBytes("name_"), StrConv::stringToBytes(this->name_.get()), this->getDBPrefix()); + dbBatch.push_back(StrConv::stringToBytes("number_"), UintConv::uint256ToBytes(this->number_.get()), this->getDBPrefix()); + dbBatch.push_back(StrConv::stringToBytes("tuple_name"), StrConv::stringToBytes(get<0>(this->tuple_)), this->getDBPrefix()); + dbBatch.push_back(StrConv::stringToBytes("tuple_number"), UintConv::uint256ToBytes(get<1>(this->tuple_)), this->getDBPrefix()); return dbBatch; } ``` diff --git a/precompiled-contracts/creating-a-protocol-contract-advanced.md b/precompiled-contracts/creating-a-protocol-contract-advanced.md index 27bc23c..3cc7d9e 100644 --- a/precompiled-contracts/creating-a-protocol-contract-advanced.md +++ b/precompiled-contracts/creating-a-protocol-contract-advanced.md @@ -40,7 +40,7 @@ Continuously monitor the performance and security of your Protocol Contract afte ## Add the Protocol Contract to the protocolContractAddresses map -Under `src/contract/contractmanager.h`, you will find a global `std::unordered_map` which contains the registered Protocol Contract addresses. You should stick to the same format as the other contracts, and add your contract address to the map. +Under `src/contract/contractmanager.h`, you will find a global map which contains the registered Protocol Contract addresses. You should stick to the same format as the other contracts, and add your contract address to the map. ## Pay attention to the contract's state diff --git a/precompiled-contracts/dynamic-and-protocol-contracts.md b/precompiled-contracts/dynamic-and-protocol-contracts.md index 297268c..b81fcd0 100644 --- a/precompiled-contracts/dynamic-and-protocol-contracts.md +++ b/precompiled-contracts/dynamic-and-protocol-contracts.md @@ -25,13 +25,20 @@ AppLayer's BDK provides ready-to-use templates for the following Dynamic Contrac * `DEXV2Router02` (template for a DEX contract router) * `UQ112x112` (namespace for dealing with fixed point fractions in DEX contracts) +Some contracts were converted directly from their OpenZeppelin counterparts and are also available for use: + +* `ERC721URIStorage` +* `Ownable` + There are also specific contracts that only exist for internal testing purposes and are not meant to be used as templates: -* `SimpleContract` (template for a simple contract, used for both testing and teaching purposes) -* `RandomnessTest` (template for testing random number generation) +* `SimpleContract` (what it says on the tin - a simple contract, used for both testing and teaching purposes) +* `RandomnessTest` (contract for testing random number generation) * `ERC721Test` (derivative contract meant to test the capabilities of the ERC721 template) * `TestThrowVars` (contract meant to test SafeVariable commit/revert functionality using exception throwing) * `ThrowTestA/B/C` (contracts meant to test nested call revert functionality) +* `SnailTracer` / `SnailTracerOptimized` (C++ conversions of the [SnailTracer](https://github.com/karalabe/snailtracer) contract, used for benchmarking purposes) +* `Pebble` (contract meant to be used in the testnet as a little NFT mining game) ## Protocol Contracts @@ -43,7 +50,8 @@ There are also specific contracts that only exist for internal testing purposes Contracts in the AppLayer network are managed by a few classes working together (check the `src/core` and `src/contract` folders for more info on each class): -* `State` (`src/core/state.h`) is responsible for owning all the Dynamic Contracts registered in the blockchain (which you can get a list of by calling the class' `getCppContracts()` and/or `getEVMContracts()` functions, depending on which ones you need), as well as their global variables (name, address, owner, balances, events, etc.) +* `State` (`src/core/state.h`) is responsible for owning all the Dynamic Contracts registered in the blockchain (which you can get a list of by calling the class' `getCppContracts()` and/or `getEVMContracts()` functions, depending on which ones you need), as well as their global variables (name, address, owner, balances, etc.) +* `Storage` (`src/core/storage.h`) is responsible for properly storing contract data, such as emitted events and the transactions that triggered them * `ContractManager` (`src/contract/contractmanager.h`) is responsible for solely creating and registering the contracts, then passing them to State (with the `ContractFactory` namespace providing helper functions to do so) * `ContractHost` (`src/contract/contracthost.h`) is responsible for allowing contracts to interact with each other and to enable them to modify balances - this kind of inter-communication is done by intercepting calls of functions from registered contracts (if the functor/signature matches), which is done through either an `eth_call` request or a transaction processed from a block * `ContractStack` (`src/contract/contractstack.h`) is responsible for managing alterations of data, like contract variables and account balances during nested contract call chains, by keeping a stack of changes made during a contract call, and automatically committing or reverting them in the account state when required (for non-view functions) @@ -63,7 +71,7 @@ Given the example Solidity contract: pragma solidity ^0.8.10; contract ExampleContract { - mapping(address => uint256) values;c + mapping(address => uint256) values; function setValue(address addr, uint256 value) external { values[addr] = value; return; @@ -71,7 +79,7 @@ contract ExampleContract { } ``` -The transpiled code should look similar to this: +The transpiled code should look similar to this (this is only an example, read further for more details): * Declaration (`ExampleContract.h`) @@ -79,14 +87,20 @@ The transpiled code should look similar to this: #include <...> class ExampleContract : public DynamicContract { private: - std::unordered_map values; + std::unordered_map values; // or boost::unordered_flat_map for example // Const-reference as they are not changed by the function. - void setValue(const Address &addr, const uint256 &value); + void setValue(const Address& addr, const uint256_t& value); public: + // Constructor for creating the contract the first time. + // "address" is where the contract will be deployed. + // "creator" is the address that created the contract. + // "chainId" is the chain ID where the contract will operate. + ExampleContract( + const Address& address, const Address& creator, const uint64_t& chainId + ); + // Constructor for loading the already-created contract from the database. ExampleContract( - const Address& contractAddress, const uint64_t& chainId, - std::unique_ptr &contractManager, - std::unique_ptr db + const Address& address, const DB& db ); void callContractWithTransaction(const Tx& transaction); } @@ -98,13 +112,19 @@ class ExampleContract : public DynamicContract { #include "ExampleContract.h" ExampleContract( - const Address& contractAddress, const uint64_t& chainId, - std::unique_ptr &contractManager, - std::unique_ptr db -) : Dynamic Contract(contractAddress, chainId, contractManager, db) { + const Address& address, const Address& creator, const uint64_t& chainId +) : DynamicContract("ExampleContract", address, creator, chainId) { // Read the "values" variables from DB - // Code generated by the transpiller from all local variables - // of the solidity contract, on the ExampleContract, you have values as a address => uint256 mapping + // Code generated by the transpiler from all local variables + // of the solidity contract, on the ExampleContract, you have values as an address => uint256 mapping + ... +} + +ExampleContract::ExampleContract( + const Address& address, + const DB& db +) : DynamicContract(address, db) { + // Same thing but the contract already exists in the database so you just load it entirely from there ... } @@ -131,28 +151,6 @@ void ExampleContract::callContractWithTransaction(const Tx& transaction) { ## The BaseContract class -The `BaseContract` class, declared in `src/contract/contract.h`, is the base class which all contracts derive from. This class holds all the [Solidity global variables](https://docs.soliditylang.org/en/v0.8.17/units-and-global-variables.html), besides variables common among these contracts (such as contract address). Its header should look similar to the following: - -```cpp -class BaseContract { - private: - // CONTRACT VARIABLES - const Address _contractAddress; - const uint64_t _chainId; - const std::unique_ptr& _contractManager; - // GLOBAL VARIABLES - static Address _coinbase; // Current Miner Address - static uint256_t _blockNumber; // Current Block Number - static uint256_t _blockTimestamp; // Current Block Timestamp - public: - Contract(const Address& contractAddress, const uint64_t& chainId, std::unique_ptr &contractManager) : _contractAddress(contractAddress), _chainId(chainId), _contractManager(contractManager) {} - const Address& coinbase() { return _coinbase }; - const uint256_t& blockNumber() { return _blockNumber}; - const uint256_t blockTimestamp() { return _blockTimestamp}; - virtual void callContractWithTransaction(const Tx& transaction); - virtual std::string ethCallContract(const std::string& calldata) const; - friend State; // State can update the private global variables of the contracts -} -``` +The `BaseContract` class, declared in `src/contract/contract.h`, is the base class which all contracts derive from. This class holds all the [Solidity global variables](https://docs.soliditylang.org/en/v0.8.17/units-and-global-variables.html), besides variables common among these contracts (such as contract address). Have a look at the header file for further reference on its structure. Regarding the `callContractWithTransaction` and the `ethCallContract` functions, `callContractWithTransaction` is used by the State when calling from `processNewBlock()`, while `ethCallContract` is used by RPC to answer for `eth_call`. Strings returned by `ethCallContract` are hex strings encoded with the desired function result. diff --git a/precompiled-contracts/safevariables-and-commit-revert-logic.md b/precompiled-contracts/safevariables-and-commit-revert-logic.md index 3ce6982..892a632 100644 --- a/precompiled-contracts/safevariables-and-commit-revert-logic.md +++ b/precompiled-contracts/safevariables-and-commit-revert-logic.md @@ -11,7 +11,7 @@ MyClass::updateValueAndThrow(const uint64_t key, const uint64_t value) { // Suppose the original value of myMap[key] is 5 this->myMap[key] = value; // e.g. 10 throw std::runtime_error("Error"); - // myMap[key] is now 10, but it shouldn't be because it threw + // myMap[key] remains 10, but it shouldn't because it threw } ``` @@ -33,15 +33,14 @@ For Protocol Contracts, that's most of the required context. Dynamic Contracts, All SafeVariables inherit from the `SafeBase` class, which adhere to the following rules: -* Have two internal variables: one for the original value and another that is a pointer to a temporary value -* Must override the `check()`, `commit()`, and `revert()` functions - * `check()` should verify if the temporary value is `nullptr`; if it is, it should set the temporary value to the original value - * `commit()` should copy the temporary value to the original value - * `revert()` should set the temporary value to `nullptr` +* Have two internal variables: one for the current/original value and another for the previous/temporary value + * Optionally, if required, an undo stack for dealing with more complex variables such as containers +* Must override the `commit()` and `revert()` functions + * `commit()` should keep the current value as-is and either discard the previous one or equal it to the current, depending on the implementation details + * `revert()` should do the opposite - discard the current value and set it back to the previous one (also applying the undo stack if it has one, again, depending on the implementation details) * Must be declared as *private* within contracts and initialized with `this` as the parameter - this enables the SafeVariable to mark itself as used in the `ContractManager`, allowing proper reversion of changes made to the contract -* Must call `check()` and return the temporary value (or reference to the temporary value) -* If non-`const`, must call `markAsUsed()` when accessed, allowing the contract to properly register the variable as used within the function call, and properly revert or commit changes made to the contract accordingly -* When initialized with values in a contract's constructor, must call `commit()` followed by `enableRegister()` so the commit/revert functionality can actually work +* Must call `markAsUsed()` on any and all non-`const` functions that it has (including ones that don't actually change the value), allowing the contract to properly register the variable as used within the function call, and properly revert or commit changes made to the contract accordingly +* When initialized with values in a contract's constructor, must call `commit()` followed by `enableRegister()` so the commit/revert functionality can actually work (see existing contracts for more info) ## Types of SafeVariables @@ -50,29 +49,17 @@ We provide the following SafeVariable types, fully functional and ready for use * `SafeAddress` - abstracts a 20-byte address * `SafeArray` - abstracts an array * `SafeBool` - abstracts a boolean +* `SafeBytes` - abstracts a raw bytes string * `SafeInt` and `SafeUint` - abstract a range of signed/unsigned integers (see Solidity ABI) * `SafeString` - abstracts a string (either literal or raw bytes) * `SafeTuple` - abstracts a tuple/struct of any type * `SafeUnorderedMap` - abstracts a mapping * `SafeVector` - abstracts a vector -## Caveats with safe containers +### Caveats -Containers have some exceptions to these rules. Copying the entire container would be prohibitively expensive, so only accessed values are copied to the temporary container. This means they do not behave like regular containers, requiring developers to exercise caution when using iterators or looping through them. - -Our `SafeUnorderedMap` variable allows limited looping through the container. This limitation is due to the inability to access both the original and temporary containers simultaneously; you can only access one at a time. It is recommended to loop through a container within a `const` function, as this will not modify the temporary container. - -`SafeUnorderedMap` includes the following functions for looping through both containers: - -| Function | Description | Return type | -| ---------- | -------------------------------------------------------------------- | --------------- | -| cbegin() | Returns a const iterator to the beginning of the original container | const\_iterator | -| cend() | Returns a const iterator to the end of the original container | const\_iterator | -| begin() | Returns a const iterator to the beginning of the temporary container | iterator | -| end() | Returns a const iterator to the end of the temporary container | iterator | -| empty() | Returns true both the original and temporary container is empty | bool | -| size() | Returns the size of the original container | size\_type | -| tempSize() | Returns the size of the temporary container | size\_type | - -Keep in mind that the temporary and original containers are not the same, so duplicates within `size()` and `tempSize()` are possible. +Some specific non-`const` functions from certain SafeVars are NOT considered "safe" at the moment - this means commit/revert logic does NOT work properly on them due to implementation issues. We advise caution for now if you want to use them, preferably in a read-only manner (as in, do not alter the value itself if using one of those): +* `SafeUnorderedMap::find()` +* `SafeUnorderedMap::begin()` +* `SafeUnorderedMap::end()` diff --git a/understanding-contracts/README.md b/understanding-contracts/README.md index 224b9e7..4bd4d49 100644 --- a/understanding-contracts/README.md +++ b/understanding-contracts/README.md @@ -6,6 +6,6 @@ description: How smart contracts work in the AppLayer protocol. Contracts in the AppLayer network are custom, developer-made classes that directly interact with the current State of the Blockchain. This chapter will comprehensively cover creating native contracts for AppLayer using the BDK, as well as operating the AppLayer EVM to leverage existing Solidity contracts. -AppLayer ensures that contracts deployed in its network, no matter the type, remain compatible with existing frontend Web3 tools (e.g. MetaMask, ethers.js, web3.js, etc.). Those are originally designed to interact with Solidity contracts and thus require a similar interface. +AppLayer ensures that contracts deployed in its network, no matter the type, remain compatible with existing frontend Web3 tools (e.g. [MetaMask](https://metamask.io/), [ethers.js](https://github.com/ethers-io/ethers.js/), [web3.js](https://docs.web3js.org/), etc.). Those are originally designed to interact with Solidity contracts and thus require a similar interface. To call your contract's functions from a frontend, you'll also need to generate its ABI - you can either do it directly with our generator tool if coding a pre-compiled contract (explained further), or replicate their definitions in Solidity and use an external tool like Ethereum's [Remix](https://remix.ethereum.org/) or any other of your preference. This ABI can then be used by your preferred Web3 frontend. diff --git a/understanding-contracts/setting-up-the-development-environment.md b/understanding-contracts/setting-up-the-development-environment.md index 33f5b38..01fbd21 100644 --- a/understanding-contracts/setting-up-the-development-environment.md +++ b/understanding-contracts/setting-up-the-development-environment.md @@ -45,7 +45,7 @@ Then, install Docker on your system (if you don't have it installed already). In * [Docker for Mac](https://docs.docker.com/docker-for-mac/install/) * [Docker for Linux](https://docs.docker.com/desktop/install/linux-install/) -Once Docker is installed, go to the root directory of your cloned repository (where the `Dockerfile` is located), and run the following command (if you're on Linux or Mac, use `sudo`): +Once Docker is installed, go to the root directory of your cloned repository (where the `Dockerfile` is located), and run the following command: ```bash docker build -t bdk-cpp-dev:latest . @@ -53,19 +53,20 @@ docker build -t bdk-cpp-dev:latest . This will build the image and tag it as `bdk-cpp-dev:latest`. You can change the tag to whatever you want, but remember to change it at the next step. -After building the image, run it with the following command (again, if using Linux or Mac, run as `sudo`): +After building the image, run a container with the following command: ```bash # For Linux/Mac -docker run -it -v $(pwd):/orbitersdk-volume -p 8080-8099:8080-8099 -p 8110-8111:8110-8111 bdk-cpp-dev:latest +docker run -it --name bdk-cpp -v $(pwd):/bdk-volume -p 8080-8099:8080-8099 -p 8110-8111:8110-8111 bdk-cpp-dev:latest # For Windows -docker run -it -v %cd%:/orbitersdk-volume -p 8080-8099:8080-8099 -p 8110-8111:8110-8111 bdk-cpp-dev:latest +docker run -it --name bdk-cpp -v %cd%:/bdk-volume -p 8080-8099:8080-8099 -p 8110-8111:8110-8111 bdk-cpp-dev:latest ``` where: +* `--name bdk-cpp` is an optional label for easier handling of the container (instead of using its ID directly) * `$(pwd)` or `%cd%` is the absolute/full path to your repository's folder -* `:/orbitersdk-volume` is the path inside the container where the BDK will be mounted. This volume is synced with the `bdk-cpp` folder inside the container +* `:/bdk-volume` is the path inside the container where the BDK will be mounted. This volume is synced with the `bdk-cpp` folder inside the container * The `-p` flags expose the ports used by the nodes - the example exposes the default ports 8080-8099 and 8110-8111, if you happen to use different ports, change them accordingly When running the container, you will be logged in as the root user and will be able to develop, build and deploy the network within the container. Remember that we are using our local repo as a volume, so every change in the local folder will be reflected to the container in real time, and vice-versa (so you can develop outside and use the container only for build and deploy). You can also integrate the container with your favorite IDE or editor. @@ -85,19 +86,31 @@ After editing the `docker-compose.yml` file, right-click on it and select `Compo ### Manual setup -You can follow these steps to build the BDK in your own system. Dependencies are: - -* **GCC** with support for **C++23** or higher -* **CMake 3.19.0** or higher -* **Boost 1.83** or higher (components: _chrono, filesystem, program-options, system, thread, nowide_) -* **OpenSSL 1.1.1** -* **CryptoPP 8.2.0** or higher -* **libscrypt** -* **zlib** -* **libsnappy** for database compression -* **tmux** (for deploying) -* (optional) **clang-tidy** for linting -* (optional) **mold** for faster/better linking +You can follow these steps to build the BDK in your own system. Dependencies are divided logically into *toolchain binaries* and *libraries*: + +* *Toolchain binaries*: + * **git** + * **GCC** with support for **C++23** or higher + * **Make** + * **CMake 3.19.0** or higher + * **Protobuf** (protoc + grpc_cpp_plugin) + * **tmux** (for deploying) + * (optional) **ninja** if you prefer it over make + * (optional) **mold** if you prefer it over ld + * (optional) **doxygen** for generating docs + * (optional) **clang-tidy** for linting +* *Libraries*: + * **Boost 1.83** or higher (components: *chrono, filesystem, program-options, system, thread, nowide*) + * **OpenSSL 1.1.1** / **libssl 1.1.1** or higher + * **libzstd** + * **CryptoPP 8.2.0** or higher + * **libscrypt** + * **libc-ares** + * **gRPC** (libgrpc and libgrpc++) + * **secp256k1** + * **ethash** + **keccak** + * **EVMOne** + **EVMC** + * **Speedb** The versions of those dependencies should suffice out-of-the-box for at least the following distros (or greater, including their derivatives): @@ -107,14 +120,23 @@ The versions of those dependencies should suffice out-of-the-box for at least th * **Fedora 40** * Any rolling release distro from around **May 2024** onwards (check their repos to be sure) -For older distros, you may need to compile some dependencies from source (specifically CMake and Boost). Make sure to uninstall them from the system first to prevent any version conflicts. +#### Tips for dependencies + +There is a script called `scripts/deps.sh` which you can use to check if you have those dependencies installed (`deps.sh --check`), install them in case you don't (`deps.sh --install`), and clean up the external ones for reinstalling (`deps.sh --cleanext`). The script expects dependencies to be installed either on `/usr` or `/usr/local`, giving preference to the latter if it finds anything there (so you can use a higher version of a dependency while still keeping your distro's default one). -#### One-liners +**Please note that installing most dependencies through the script only works on APT-based distros** (Debian, Ubuntu and derivatives) - you can still check the dependencies on any distro and install the few ones labeled as "external" (those are fetched through `git`), but if you're on a distro with another package manager and/or a distro older than one of the minimum ones listed above, you're on your own. -* For APT-based distros: +For Debian specifically, you can (and should) use `update-alternatives` to register and set your GCC version to a more up-to-date build if required. + +If you're using a self-compiled GCC build out of the system path (e.g. `--prefix=/usr/local/gcc-X.Y.Z` instead of `--prefix=/usr/local`), don't forget to export its installation paths in your `PATH` and `LD_LIBRARY_PATH` env vars (to prevent e.g. "version `GLIBCXX_...'/`CXXABI_...` not found" errors). Put something like this in your `~/.bashrc` file for example, changing the version accordingly to whichever one you have installed: ```bash -sudo apt install git build-essential cmake mold tmux clang-tidy autoconf libtool pkg-config libboost-all-dev libcrypto++-dev libscrypt-dev libsnappy-dev libssl-dev zlib1g-dev openssl +# For GCC in /usr/local +export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH + +# For self-contained GCC outside /usr/local +export PATH=/usr/local/gcc-14.2.0/bin:$PATH +export LD_LIBRARY_PATH=/usr/local/gcc-14.2.0/lib64:$LD_LIBRARY_PATH ``` ## Compiling @@ -145,26 +167,15 @@ make -j$(nproc) cmake --build . -- -j$(nproc) ``` -After building, you can optionally run a test bench with the following command: `./src/bins/orbitersdkd-tests/orbitersdkd-tests -d yes` (the `-d yes` parameter will give a verbose output). You can also use filter tags to test specific parts of the project (e.g. `./src/bins/orbitersdkd-tests/orbitersdkd-tests [utils] -d yes` will test all the components inside the `src/utils` folder, `[utils][tx]` will test only the transaction-related components inside utils, etc.). +Adjust `-j$(nproc)` accordingly to your system if necessary, as some parts of the project can get really heavy RAM-wise during compilation. -Usable tags are: +After building, you can optionally run a test bench with the following command: `./src/bins/bdkd-tests/bdkd-tests -d yes` (the `-d yes` parameter will give a verbose output). -* `[utils]` - everything in `src/utils` - * `[utils][*]` - replace `*` with one of the class names: `block`, `db`, `hex`, `merkle`, `randomgen`, `secp256k1`, `strings`, `tx` (TxBlock), `txvalidator`, `utilsitself` (the last one refers to the actual Utils class), `[*][throw]` (for testing actual throwing conditions) -* `[contract]` - everything in `src/contract` - * `[contract][*]` - replace `*` with one of the contract names: `abi`, `contractabigenerator`, `contractmanager`, `dexv2`, `erc20`, `erc20wrapper`, `nativewrapper` - * `[contract][variables][*]` - all SafeVariable types - `[*]` is optional, if you want to test specific variables, replace `*` with one of the class names: `safeaddress`, `safearray`, `safebool`, `safestring`, `safeuintX_t` (replace X with a uint size from 8 to 256), `safeunorderedmap`, `safevector` -* `[core]` - everything in `src/core` - * `[core][*]` - replace `*` with one of the class names: `blockchain`, `options`, `rdpos`, `state`, `storage` - * `[core][rdpos][net]` - only networking-related functionality - append `[heavy]` for very taxing functionality -* `[net]` - everything in `src/net` - * `[net][http][jsonrpc]` - only JSONRPC-related functionality -* `[p2p]` - only P2P-related functionality -* `[sdktestsuite]` - test the SDKTestSuite itself (test suite used internally for testing contracts) +You can also use filter tags to test specific parts of the project (e.g. `./src/bins/bdkd-tests/bdkd-tests -d yes [utils]` will test all the components inside the `src/utils` folder, `[utils][tx]` will test only the transaction-related components inside utils, etc.). You can check all the available tags by doing a `grep -rw "\"\[.*\]\""` in the `tests` subfolder. ## Deploying -Currently there are two ways to deploy an AppLayer node: _manual_, and _dockerized_. Go back to the project's root folder and check the `scripts` subfolder - there are two main scripts there used for deploying the node. You can pick whichever one you prefer, depending on your needs. +Currently there are two ways to deploy an AppLayer node: *dockerized* and *manual*. Go back to the project's root folder and check the `scripts` subfolder - there are two main scripts there used for deploying the node. You can pick whichever one you prefer, depending on your needs. ### Dockerized deploy @@ -204,4 +215,4 @@ As an example, here's how to configure MetaMask to connect to your local testnet
-Once you're connected, import the following private key for the chain owner account: `0xe89ef6409c467285bcae9f80ab1cfeb3487cfe61ab28fb7d36443e1daa0c2867`. This account contains 1000 APPL Tokens from the get go and is able to call the `ContractManager` contract. +Once you're connected, import the following private key for the chain owner account: `0xe89ef6409c467285bcae9f80ab1cfeb3487cfe61ab28fb7d36443e1daa0c2867`. This account contains 1000 APPL Tokens from the get go and is able to call the `ContractManager` contract. Other details about the deployed testnet chain can be found in the project's README.md file. diff --git a/understanding-contracts/solidity-abi.md b/understanding-contracts/solidity-abi.md index 2a8542e..b4d8e10 100644 --- a/understanding-contracts/solidity-abi.md +++ b/understanding-contracts/solidity-abi.md @@ -8,7 +8,7 @@ description: >- AppLayer is primarily a _native, pre-compiled_ blockchain, which means its main focus is to run without the need for an EVM. However, the vast majority of the smart contract ecosystem operates and depends on [Solidity](https://docs.soliditylang.org/en/latest) - not only the contracts themselves but also the data they share across each other. -When developing pre-compiled contracts, AppLayer makes use of an abstraction of Solidity's ABI encoding and decoding processes to properly translate between native and non-native data types. The **ABI** namespace, declared in `src/contract/abi.h`, contains several functions for Solidity ABI-related operations, for managing and manipulating data in Solidity format. +When developing pre-compiled contracts, AppLayer makes use of an abstraction of Solidity's ABI encoding and decoding processes to properly translate between native and non-native data types. The **ABI** namespace (`src/contract/abi.h`) contains several functions for Solidity ABI-related operations, for managing and manipulating data in Solidity format. This is only an overview, check the [Doxygen](https://doxygen.nl) docs for more details on how those functions work. @@ -16,8 +16,6 @@ This is only an overview, check the [Doxygen](https://doxygen.nl) docs for more The **Types** enum contains the supported Solidity data types in the ABI. Each value has an intrinsic equivalency with both the Solidity data type and the native C++ data type that it represents. -TODO: gone? -> **BaseTypes** is a `std::variant` declared in `src/utils/utils.h` that abstracts all of the types in one typedef, for easier handling. - Replace the **X** in "uintX" and "intX" with the desired size number. The ABI supports every size from 8 to 256 (inclusive), in multiples of 8 (e.g. 8, 16, 24, 32, 40, 48, ..., until 256) - in other words, `x <= 256 && x % 8 == 0`. **Enums are encoded as uint8**. | Enum | Solidity | C++ | diff --git a/understanding-rdpos/README.md b/understanding-rdpos/README.md index 68cb5ea..5384fe1 100644 --- a/understanding-rdpos/README.md +++ b/understanding-rdpos/README.md @@ -4,6 +4,6 @@ description: How AppLayer uses Random Deterministic Proof of Stake. # Understanding rdPoS -To keep consensus on such a blazing fast network without tripping up and/or having to deal with rollbacks, AppLayer uses *random deterministic block creation* that allows only one given node to create a block for a given time, eliminating the risk of a block race condition in the network. +To keep consensus on such a blazing fast network without tripping up and/or having to deal with rollbacks, AppLayer uses a *random deterministic block creation* that allows only one given node to create a block for a given time, eliminating the risk of a block race condition in the network. This is implemented in AppLayer as its own consensus algorithm, called **rdPoS (Random Deterministic Proof of Stake)**, which empowers Validators and Sentinels to deal with block congestion and random number generation. This chapter aims to explain in-depth the rdPoS algorithm used by the AppLayer protocol, from concept to implementation. diff --git a/understanding-rdpos/slashing.md b/understanding-rdpos/slashing.md index b21f65b..da5f685 100644 --- a/understanding-rdpos/slashing.md +++ b/understanding-rdpos/slashing.md @@ -4,7 +4,7 @@ description: Cutting out malicious nodes, carefully. # Slashing -What happens when a validator answers with a "randomness" hash that does not match its own hash? Or when a validator creates an invalid block with invalid transactions? Or when a validator can't create a block before reaching the network's time limit? +What happens when a Validator answers with a "randomness" hash that does not match its own hash? Or when a Validator creates an invalid block with invalid transactions? Or when a Validator can't create a block before reaching the network's time limit? Misbehaving nodes suffer consequences. Since Validator signatures are required at protocol level, if a Validator tries to break the rules, it's possible to know who it is thanks to the signature, and "slash" it from the network.