diff --git a/src/RFC-0110_BaseNodes.md b/src/RFC-0110_BaseNodes.md index 9b87be5..aa5d7b5 100644 --- a/src/RFC-0110_BaseNodes.md +++ b/src/RFC-0110_BaseNodes.md @@ -172,7 +172,7 @@ Pseudocode: output_weight += serialize(output.script) / BYTES_PER_GRAM output_weight += serialize(output.features) / BYTES_PER_GRAM - input_weight = num_inputs * PER_INPUT_GRAMS(9) + input_weight = num_inputs * PER_INPUT_GRAMS(8) kernel_weight = num_kernels * PER_KERNEL_GRAMS(10) weight = output_weight + input_weight + kernel_weight diff --git a/src/RFC-0111_BaseNodeArchitecture.md b/src/RFC-0111_BaseNodeArchitecture.md index 7f9570e..cdd33e2 100644 --- a/src/RFC-0111_BaseNodeArchitecture.md +++ b/src/RFC-0111_BaseNodeArchitecture.md @@ -147,16 +147,17 @@ This module is used by the Base Node Service, Base Node State Machine, Mempool S this is delivered using the Lightning Memory-mapped Database (LMDB). LMDB is highly performant, intelligent and straightforward to use. An LMDB is essentially treated as a hash map data structure that transparently handles memory caching, disk Input/Output (I/O) and multi-threaded access. This module is shared by many services and so must -be thread-safe. +be thread-safe. The current UTXO set is maintained as a Jellyfish Merkle Tree (JMT) — a sparse Merkle tree — backed by +the LMDB store, whose root is committed to in the block header as `output_mr`. ## Communication Interfaces ### P2P communications The Tari Peer to Peer messaging protocol is defined in [RFC-0172]. It is a fire-and-forget style protocol. Messages can -be sent directly to a known peer, sent indirectly to an offline or unknown peer and broadcast to a set of peers. When -a message is sent to a specific peer it is propagated to the peer's local neighbourhood and stored by those peers until it -comes online to receive the message. Messages that are broadcast will be propagated around the network until the whole -network has received them, they are not stored. +be sent directly to a known peer or broadcast to a set of peers. Messages that are broadcast will be propagated around +the network until the whole network has received them; they are not stored. (Store-and-forward messaging, which +previously allowed messages to be stored for offline peers, has been removed; all current nodes discard store-and-forward +messages.) ### RPC Services Fire-and-forget messaging is not efficient for point to point communications between online peers. For these applications diff --git a/src/RFC-0120_Consensus.md b/src/RFC-0120_Consensus.md index 695df45..cc579d0 100644 --- a/src/RFC-0120_Consensus.md +++ b/src/RFC-0120_Consensus.md @@ -131,7 +131,7 @@ A coinbase transaction contained in a block MUST: * MUST NOT have a script offset other than 0 A coinbase transaction contained in a block MAY: -* include any arbitrary 64 bytes of extra data; see [coinbase-extra] +* include any arbitrary extra data up to 256 bytes (as limited by `coinbase_output_features_extra_max_length`); see [coinbase-extra] ### Block Headers [block headers]: #block-headers "Block headers" @@ -146,13 +146,12 @@ Every [block header] MUST contain the following fields: * output_mr; * block_output_mr; * output_smt_size; -* witness_mr; * kernel_mr; * kernel_mmr_size; * total_kernel_offset; -* script_kernel_offset; +* total_script_offset; * validator_node_mr; -* validator_size; +* validator_node_size; * pow; * nonce. @@ -215,17 +214,17 @@ The output_mr MUST conform to the following: * Represented as an array of unsigned 8-bit integers (bytes) in little-endian format. * The hashing function used MUST be blake2b with a 256-bit digest. -#### Output_mmr_size +#### Output_smt_size -This is the total size of the leaves in the output Merkle mountain range. +This is the total number of leaves in the output sparse Merkle tree (Jellyfish Merkle Tree), which tracks the current UTXO set. -The Output_mmr_size MUST conform to the following: +The Output_smt_size MUST conform to the following: * Represented as a single unsigned 64-bit integer. #### Block_output_mr -The `block_output_mr` MUST be calculated as the Merkle mountain range tree root of all non-coinbase outputs, appended with the Merkle mountain range tree root of all coinbase outputs. +The `block_output_mr` MUST be calculated as the Merkle mountain range root of a structure that contains, as leaves, the hash of each coinbase output in canonical order, followed by the Merkle mountain range root of all non-coinbase outputs. The block_output_mr MUST conform to the following: @@ -283,7 +282,7 @@ This is the Nonce used in solving the Proof of Work. The Nonce MUST conform to the following: * MUST be transmitted as an unsigned 64-bit integer. -* For RandomX blocks, this MUST be 0. +* For RandomX-M (Monero merge-mined) blocks, this MUST be 0. RandomX-T blocks may use the nonce as part of the PoW solution. #### PoW @@ -291,10 +290,12 @@ This is the Proof of Work algorithm used to mine the block. It is used in conjun The [PoW] MUST contain the following: -* pow_algo as an enum (0 for RandomX-M, 1 for Sha3x, 2 for RandomX-T, and 3 for C29). -* pow_data for RandomX blocks as an array of unsigned 8-bit integers (bytes) in little-endian format, containing the RandomX merge-mining Proof-of-Work data. - * The RandomX seed, stored as `randomx_key` within the RandomX block, MUST NOT have been first seen in a block with more than `max_randomx_seed_height` confirmations. +* pow_algo as an enum (0 for RandomX-M, 1 for Sha3x, 2 for RandomX-T, and 3 for Cuckaroo, also known as C29). +* pow_data for RandomX-M (Monero merge-mined) blocks as an array of unsigned 8-bit integers (bytes) in little-endian format, containing the Monero merge-mining Proof-of-Work data. + * The RandomX seed, stored as `randomx_key` within the RandomX-M block, MUST NOT have been first seen in a block with more than `max_randomx_seed_height` confirmations (3000 blocks on mainnet). +* pow_data for RandomX-T (Tari solo RandomX) blocks MUST be at most 32 bytes. * pow_data for Sha3x blocks MUST be empty. +* pow_data for Cuckaroo blocks MUST be exactly `ceil(cuckaroo_cycle_length * cuckaroo_edge_bits / 8)` bytes (153 bytes on mainnet where cycle length=42 and edge bits=29), containing the packed cuckaroo cycle edges. The last byte MUST be zero-padded if the total bit count is not a multiple of 8. #### Difficulty Calculation [target difficulty]: #target-difficulty "Target Difficulty" @@ -310,8 +311,8 @@ $$ | Symbol | Value | Description | |-------------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------| | N | 90 | Target difficulty block window | -| T | 480 | Target block time in seconds. The value used depends on the PoW algorithm being used. | -| \\( \solvetimemax \\) | 2880 | Maximum solve time. This is six times the target time of the current PoW algorithm. | +| T | 480 (current) | Target block time in seconds per PoW algorithm. The value used depends on the PoW algorithm and the block height. On mainnet: 240s for heights 0–14,999; 360s for heights 15,000–94,999; 480s from height 95,000 onward. | +| \\( \solvetimemax \\) | 6 × T | Maximum solve time. This is six times the target time of the current PoW algorithm (e.g. 2880s when T=480, 2160s when T=360, 1440s when T=240). | | \\( \solvetime \\) | variable | The timestamp difference in seconds between block _i_ and _i - 1_ where \\( 1 \le \solvetime \le \solvetimemax \\) | | \\( \mathrm{D_{avg}} \\)| variable | The average difficulty of the last _N_ blocks | @@ -324,31 +325,30 @@ $$ \tag{2} $$ -It is important to note that the two proof-of-work algorithms are calculated _independently_; i.e., if the current block uses _SHA3x_ proof of work, the block window and solve times only include _SHA3x_ blocks, and vice versa. +It is important to note that the proof-of-work algorithms are calculated _independently_; i.e., if the current block uses _SHA3x_ proof of work, the block window and solve times only include _SHA3x_ blocks. The same applies to each of the other algorithms (RandomX-M, RandomX-T, and Cuckaroo). ### FTL [FTL]: #ftl "Future Time Limit" The Future Time Limit (FTL) defines how far into the future a timestamp is accepted as valid. Any block with a timestamp beyond the FTL is rejected until the current time catches up. -The FTL is calculated as (T*N)/20, with T and N defined as: -- T: Target time — the ideal time that should pass between mined blocks. -- N: Block window — the number of blocks used when calculating difficulty adjustments. +The FTL is hardcoded per consensus version. On mainnet it is 540 seconds. The design formula is (T_chain*N)/20, where T_chain is the overall chain target block time (the ideal time between any two consecutive blocks, which equals the per-algorithm target time divided by the number of active PoW algorithms), and N is the difficulty block window (90). For mainnet, T_chain = 120s (e.g. 240s/2 algos or 480s/4 algos), so FTL = 120*90/20 = 540. Note that T_chain is distinct from the per-algorithm target time T used in the difficulty calculation table. ### MTP [MTP]: #mtp "Median Time Passed" -The Median Time Passed (MTP) is the lower bound calculated by taking the median timestamp of the last _N_ blocks. Any block with a timestamp less than the MTP will be rejected. +The Median Time Passed (MTP) is the lower bound calculated by taking the median timestamp of the last `median_timestamp_count` blocks (11 blocks on mainnet). This is a separate consensus constant from the difficulty block window _N_. Any block with a timestamp less than or equal to the MTP will be rejected. ### Total accumulated proof of work -This is defined as the total accumulated proof of work done on the blockchain. Tari uses four _independent_ proof-of-work algorithms rated at different difficulties. To compare them, they are simply multiplied together into one number: +This is defined as the total accumulated proof of work done on the blockchain. Tari uses up to four _independent_ proof-of-work algorithms rated at different difficulties. To compare them, their accumulated difficulties are multiplied together into one number. The total accumulated difficulty is computed as the product of the accumulated RandomX-M, RandomX-T, and SHA3x difficulties. The accumulated Cuckaroo (C29) difficulty is additionally included in the product only when the consensus flag `include_c29_accumulated_difficulty_into_total` is set (on mainnet, this is true from block height 126,000 onward): $$ \begin{align} - \textit{accumulated_randomxM_difficulty} * \textit{accumulated_sha3x_difficulty} * \textit{accumulated_randomxT_difficulty} * \textit{accumulated_C29_difficulty} + \textit{total} = \textit{accumulated\_randomxM} \times \textit{accumulated\_randomxT} \times \textit{accumulated\_sha3x} \quad [\times \textit{accumulated\_cuckaroo} \text{ if enabled}] \end{align} \tag{3} $$ +Algorithms that have not yet been activated have an accumulated difficulty of 1 (the minimum), so they do not affect the product. This value is used to compare chain tips to determine the strongest chain. ### Transaction Ordering diff --git a/src/RFC-0131_Mining.md b/src/RFC-0131_Mining.md index ee871f9..a8cf90a 100644 --- a/src/RFC-0131_Mining.md +++ b/src/RFC-0131_Mining.md @@ -121,7 +121,7 @@ and it proved to be a good choice in the multi-PoW scene as well. Tari's proof-of-work mining algorithm is summarized below: - Four mining algorithms, with an average combined target block time of 120 s, to match Monero's block interval. -- A log-weighted moving average difficulty adjustment algorithm using a window of 90 blocks. +- A linear weighted moving average (LWMA) difficulty adjustment algorithm using a window of 90 blocks. ### Tari mining hash @@ -136,14 +136,16 @@ First, the block header is hashed with the 256-bit Blake2b hashing algorithm usi - input Merkle root - output Merkle root - output Merkle mountain range size -- witness Merkle root +- block output Merkle root - kernel Merkle root - kernel Merkle mountain range size - total kernel offset - total script offset +- validator node Merkle root +- validator node Merkle mountain range size This hash is used in both the SHA-3 and RandomX proof-of-work algorithms. The header version for the Tari Genesis -block is 1. +block is 0. #### RandomX ##### RxM @@ -153,7 +155,7 @@ Monero blocks that are merge-mining Tari MUST include the Tari mining hash in th Tari also imposes the following consensus rules: - The `seed_hash` MUST only be used for 3000 blocks, after which a block MUST be discarded if it's used again. - The little-endian difficulty MUST be equal to or greater than the target for that block as determined by the LWMA for Tari. -- The LWMA MUST use a target time of 800 seconds. +- The LWMA MUST use a target time of 480 seconds. - MUST set the header field PoW:pow_algo as 0 for a RxM block - MUST encode the following data into the Pow:Pow_data field: - Monero BlockHeader, @@ -168,7 +170,7 @@ Using RandomX solely with Tari, the following consensus is enforced: - The `seed_hash` MUST only be used for 3000 blocks, after which a block MUST be discarded if it's used again. - The little-endian difficulty MUST be equal to or greater than the target for that block as determined by the LWMA for Tari. -- The LWMA MUST use a target time of 800 seconds. +- The LWMA MUST use a target time of 480 seconds. - MUST set the header field PoW:pow_algo as 2 for an RxT block - MUST encode the following data into the Pow:Pow_data field: - 32 bytes extra data @@ -195,7 +197,7 @@ Tari imposes the following consensus rules: LWMA for Tari. The difficulty and target are related by the equation `difficulty = (2^256 - 1) / target`. - MUST set the header field PoW:pow_algo as 1 for a Sha block. - The PoW:pow_data field is empty -- The LWMA MUST use a target time of 800 seconds. +- The LWMA MUST use a target time of 480 seconds. A triple hash is selected to keep the requirements on hardware miners (FPGAs, ASICs) fairly low. But we also want to avoid making the proof-of-work immediately "NiceHashable". There are several coins that already use a single or @@ -212,8 +214,8 @@ expectation in block explorers and elsewhere that block hashes should always sta #### C29 Using C29 with Tari, the following consensus is enforced: -- The little-endian difficulty MUST be equal to or greater than the target for that block as determined by the LWMA for Tari. -- The LWMA MUST use a target time of 800 seconds. +- The big-endian difficulty MUST be equal to or greater than the target for that block as determined by the LWMA for Tari. +- The LWMA MUST use a target time of 480 seconds. - MUST set the header field PoW:pow_algo as 3 for a C29 block - MUST encode the following data into the Pow:Pow_data field: - Cuckaroo 29 Pow data bits diff --git a/src/RFC-0132_Merge_Mining_Monero.md b/src/RFC-0132_Merge_Mining_Monero.md index 1ba504e..1cf4dd1 100644 --- a/src/RFC-0132_Merge_Mining_Monero.md +++ b/src/RFC-0132_Merge_Mining_Monero.md @@ -65,7 +65,7 @@ This document describes the specific protocol Tari uses to merge mine with Moner ## Introduction -Tari employs a hybrid mining strategy, accepting 2 mining algorithms whose difficulties are independent of each other as discussed in [RFC-0131_Mining.html](/RFC-0131_Mining.html). +Tari employs a hybrid mining strategy, accepting four mining algorithms whose difficulties are independent of each other as discussed in [RFC-0131_Mining.html](/RFC-0131_Mining.html). This RFC details a protocol to enable Tari to accept Monero proof of work, enabling participating miners a chance to produce a valid block for either or both chain without additional mining effort. The protocol must enable a Tari base node to make the following assertions: @@ -84,9 +84,9 @@ are independent (See [RFC-0131_Mining.html](/RFC-0131_Mining.html)). Next, a coi the `get_coinbase` gRPC function. Next, the coinbase transaction is added to the new block template and passed back to the base node for the new MMR roots to be calculated. -Furthermore, the base node constructs a _Blake256_ hash of _some_ of the Tari header fields. We'll call this hash the merge mining hash \\( h_m \\) that commits to -the following header fields in order: `version`, `height`,`prev_hash`,`timestamp`,`input_mr`, `output_mr`,`output_mmr_size`,`kernel_mr`, -`kernel_mmr_size`,`total_kernel_offset`,`total_script_offset`. Note, this hash does not include the `pow` and `nonce` fields, as these fields are set as part of mining. +Furthermore, the base node constructs a _Blake2b-256_ hash of the Tari header fields (excluding the `nonce` and `pow` fields). We'll call this hash the merge mining hash \\( h_m \\) that commits to +the following header fields in order: `version`, `height`, `prev_hash`, `timestamp`, `input_mr`, `output_mr`, `output_mmr_size`, `block_output_mr`, `kernel_mr`, +`kernel_mmr_size`, `total_kernel_offset`, `total_script_offset`, `validator_node_mr`, `validator_node_size`. Note, this hash does not include the `pow` and `nonce` fields, as these fields are set as part of mining. To have the chance of mining a Monero block as well as a Tari block, we must obtain a new valid monero block template, by calling [get_block_template]. This returns a `blocktemplate_blob`, that is, a serialized Monero block containing the Monero block header, coinbase and a list of hashes referencing the @@ -111,22 +111,26 @@ reconstructed. A rust port of [Monero's tree hash] algorithm is needed to achiev This satisfies **REQ 2**, proving that the proof-of-work was performed for the Tari block. The block may now be mined. Once a solution is found that satisfies the Tari difficulty, the miner must include enough data to allow the Tari blockchain to assert **REQ 1** and **REQ 2**. -Concretely, A miner must serialize `MoneroPowData` using Monero consensus encoding and add it to the `pow_data` field in the Tari header. +Concretely, A miner must serialize `MoneroPowData` using Borsh and add it to the `pow_data` field in the Tari header. ```rust,ignore pub struct MoneroPowData { /// Monero header fields - header: MoneroBlockHeader, - /// randomX vm key - randomx_key: FixedByteArray, // Fixed 64 bytes - /// transaction count - transaction_count: u16, - /// transaction root - merkle_root: MoneroHash, + pub header: MoneroBlockHeader, + /// RandomX VM key (variable length up to 60 bytes, stored in a 64-byte container) + pub randomx_key: FixedByteArray, + /// The number of transactions included in this Monero block + pub transaction_count: u16, + /// Transaction root + pub merkle_root: MoneroHash, /// Coinbase merkle proof hashes - coinbase_merkle_proof: MerkleProof, - /// Coinbase tx from Monero - coinbase_tx: MoneroTransaction, + pub coinbase_merkle_proof: MerkleProof, + /// Incomplete Keccak hash state of the coinbase transaction prefix + pub coinbase_tx_hasher: Keccak, + /// Extra field of the coinbase transaction + pub coinbase_tx_extra: RawExtraField, + /// Aux-chain merkle proof hashes (for multi-chain / AuxPoW merge mining) + pub aux_chain_merkle_proof: MerkleProof, } ``` _fig 2. Monero PoW data struct serialized in Tari blocks_ @@ -134,7 +138,6 @@ _fig 2. Monero PoW data struct serialized in Tari blocks_ ```rust,ignore pub struct MerkleProof { branch: Vec, - depth: u16, path_bitmap: u32, } ``` @@ -143,7 +146,7 @@ _fig 3. Merkle proof struct_ A verifier may now check that the `coinbase_tx` contains the merge mining hash \\( h_m \\), and validate the `coinbase_merkle_proof` against the `transaction_root`. The `coinbase_merkle_proof` contains the minimal proof required to construct the `transaction_root`. -For example, a proof for a merkle tree of 4 hashes will require 2 hashes (h_1, h_23) of 32 bytes each, 4 bytes for the path bitmap and 2 bytes for the depth. +For example, a proof for a merkle tree of 4 hashes will require 2 hashes (h_1, h_23) of 32 bytes each and 4 bytes for the path bitmap. ```text Root* / \ @@ -156,9 +159,10 @@ For example, a proof for a merkle tree of 4 hashes will require 2 hashes (h_1, h ## Serialisation -For Monero proof-of-work, Monero consensus encoding MUST be used to serialize the `MoneroPowData` struct. Given the same inputs, -this encoding will byte-for-byte the same. The encoding uses VarInt for all integer types, allowing byte-savings, in particular -for fields that typically contain small values. Importantly, extra bytes that a miner _could_ tack onto the end of the `pow_data` field +For Monero proof-of-work, the `MoneroPowData` struct is serialized using Borsh. Given the same inputs, +this encoding will be byte-for-byte the same. Some nested Monero types (the block header and merkle root) use Monero +consensus encoding (which uses VarInt for those types), while scalar fields such as `transaction_count` use Borsh's +fixed-size encoding. Importantly, extra bytes that a miner _could_ tack onto the end of the `pow_data` field are expressly disallowed. ## Merge Mining Proxy @@ -191,7 +195,7 @@ Tari block is retrieved, enriched with the `MoneroPowData` struct and submitted [Monero's tree hash]: https://github.com/monero-project/monero/blob/1c8e598172bd2eddba2607cae0804db2e685813b/src/crypto/tree-hash.c [merge mining subfield]: https://docs.rs/monero/0.13.0/monero/blockdata/transaction/enum.SubField.html#variant.MergeMining -[Tari merge mining proxy]: https://github.com/tari-project/tari/blob/development/applications/tari_merge_mining_proxy +[Tari merge mining proxy]: https://github.com/tari-project/tari/blob/development/applications/minotari_merge_mining_proxy [get_block_template]: https://ww.getmonero.org/resources/developer-guides/daemon-rpc.html#get_block_template [bincode]: https://docs.rs/bincode/1.3.3/bincode/ [Monero daemon RPC]: https://www.getmonero.org/resources/developer-guides/daemon-rpc.html diff --git a/src/RFC-0140_Syncing_and_seeding.md b/src/RFC-0140_Syncing_and_seeding.md index 56e1ab3..6a78583 100644 --- a/src/RFC-0140_Syncing_and_seeding.md +++ b/src/RFC-0140_Syncing_and_seeding.md @@ -67,21 +67,25 @@ This Request for Comment (RFC) describes the syncing and pruning process. ### Syncing When a new node comes online, loses connection or encounters a chain reorganization that is longer than it can tolerate, -it must enter syncing mode. This will allow it to recover its state to the newest up-to-date state. Syncing can be -divided into two [SynchronizationStrategy]s: complete sync and horizon sync. Complete sync means that the node -communicates with an archive node to get the complete history of every single block from genesis block. Horizon Sync -involves the node getting every block from its [pruning horizon] to [current head], as well as every block header -up to the genesis block. +it must enter syncing mode. This will allow it to recover its state to the newest up-to-date state. The base node runs +as a finite state machine with the following states: `Starting`, `Listening`, `HeaderSync`, `DecideNextSync`, +`HorizonStateSync`, `BlockSync`, `Waiting`, and `Shutdown`. Sync failures (header, horizon, or block) transition the +node to a `Waiting` backoff state (default 5 s) before returning to `Listening`. Syncing can be divided into two +[SynchronizationStrategy]s: complete sync and horizon sync. The strategy is not chosen up front; after header sync the +`DecideNextSync` state selects it based on the local `pruning_horizon` and peer metadata. Complete sync means that the node +communicates with an archive node (one whose `pruned_height == 0`) to get the complete history of every single block from +genesis block. Horizon Sync is used by pruned nodes whose local tip is below the new horizon height; it fetches the kernel +and UTXO state up to `chain_tip − pruning_horizon`, along with every block header from genesis to the current tip. To determine if the node needs to synchronise, the node will monitor the broadcasted `chain_metadata` messages provided by its neighbours. The fields in the `chain_metadata` messages MUST be: | Field | Description | | --- | --- | -| height_of_longest_chain | 64-bit unsigned | -| best_block | hash of the tip block (32-bit) | +| best_block_height | 64-bit unsigned | +| best_block_hash | hash of the tip block (256-bit / 32 bytes) | | pruning_horizon | 64-bit unsigned | | pruned_height | 64-bit unsigned | -| accumulated_difficulty | 128-bit unsigned | +| accumulated_difficulty | 512-bit unsigned (split into 256-bit high/low halves on the wire) | | timestamp | 64-bit unsigned | #### Complete Sync @@ -100,11 +104,14 @@ The syncing process MUST be done in the following steps: 2. Sync all missing headers from the genesis block to the current chain tip. The initial header sync allows the node to confirm that the syncing peer does indeed have a fully intact chain from which to sync that adheres to this node's consensus rules and has a valid proof-of-work that is higher than any competing chains. -3. Set [SynchronizationState] to `block_sync`. -4. Start downloading blocks from sync peer starting with the oldest block in our database. A fresh node will start from - the genesis block. -5. Download all block up to [current head], validating and adding the blocks to the local chain storage as we go. -6. Once all blocks have been downloaded up and including the current network tip set the [SynchronizationState] to +3. Set [SynchronizationState] to `decide_next_sync`. This state inspects the local and peer chain metadata and + selects the appropriate follow-up strategy: it emits `ProceedToBlockSync`, `ProceedToHorizonSync`, or returns to + `listening` (`Continue`) if no suitable peer is found. +4. Set [SynchronizationState] to `block_sync`. +5. Start downloading blocks from the sync peer, beginning at the local best (tip) block and streaming forward to the + synced header tip. A fresh node (whose best block is the genesis block) will therefore start from the genesis block. +6. Download all blocks up to [current head], validating and adding the blocks to the local chain storage as we go. +7. Once all blocks have been downloaded up and including the current network tip set the [SynchronizationState] to `listening`. After this process, the node will be in sync, and will be able to process blocks and transactions normally as they @@ -119,14 +126,20 @@ The horizon sync process MUST be done in the following steps: confirm that the syncing peer does indeed have a fully intact chain from which to sync that adheres to this nodes consensus rules and has a valid proof-of-work that is higher than any competing chains. 3. Set [SynchronizationState] to `horizon_sync`. -4. Download all kernels from the current network tip back to this node's [pruning horizon]. -5. Validate kernel MMR root against headers. -6. Download all [utxo]'s from the current network tip back to this node's [pruning horizon]. -7. Validate outputs and [utxo] MMR. -8. Validate the chain balances with the expected total emission that the final sync height. -9. Once all kernels and [utxo]s have been downloaded from the network tip back to this node's [pruning horizon] set - the [SynchronizationState] to `block_sync`. This hands over further syncing to the standard sync protocol which - should return to the `listening` state if no further data has been received from peers. +4. Download all transaction kernels forwards from the local kernel MMR position up to this node's [pruning horizon] + height (`chain_tip − pruning_horizon`). +5. Validate the kernel MMR root against each header as the sync progresses. +6. Download all [utxo]'s (and spent-output markers) forwards from genesis (or the last checkpoint) up to this node's + [pruning horizon] height, in independently-verifiable tranches. +7. Validate outputs and the UTXO Jellyfish Merkle Tree (JMT / sparse Merkle tree) root against the horizon sync + header's `output_mr`. +8. Validate the chain balances at the final sync height: the sum of UTXO commitments plus burned-output commitments + must equal the expected total emission commitment plus the total kernel excess sum plus the accumulated kernel offset. +9. Once all kernels and [utxo]s have been downloaded up to this node's [pruning horizon] height, finalize the + horizon state: set the local best block, set `pruned_height` to the horizon sync height, store the horizon data + (kernel and UTXO commitment sums), and clear the output-sync checkpoint. Then set the [SynchronizationState] to + `block_sync`. This hands over further syncing to the standard sync protocol which should return to the `listening` + state if no further data has been received from peers. After this process, the node will be in sync, and will be able to process blocks and transactions normally as they arrive. @@ -138,9 +151,12 @@ that they are alive. When a node sends a ping message, it MUST include the all t receiving node MUST reply with a pong message, which should also include its version of the `chain_metadata` information. When a node receives pong replies from the current ping round, or the timeout expires, the collected `chain_metadata` -replies will be examined to determine what the current best chain is, i.e. the chain with the most accumulated work. -If the best chain is longer than out chain data the node will set [SynchronizationState] to `header_sync` and catch up -with the network. +replies will be examined to determine what the current best chain is, i.e. the chain with the most accumulated work +(highest accumulated difficulty, a 512-bit value). If a peer advertises a higher accumulated difficulty than the local +chain, the node first enters a `BehindButNotYetLagging` state to give block propagation a chance to deliver the block +inline; it only transitions to `header_sync` once it is more than `blocks_behind_before_considered_lagging` (default 1) +blocks behind, or once `time_before_considered_lagging` (default 10 s) has elapsed since the stronger chain was first +seen. #### Chain Forks @@ -149,7 +165,8 @@ will detect that it has fallen behind other nodes in the network. It will then p header sync process will be able to detect that a chain fork has occurred. The header sync process will then determine which chain is the correct chain with the highest accumulated work. If required this node will switch the best chain and proceed to sync the new blocks required to catch up to the correct chain. This process is called a chain -reorganization or [reorg]. +reorganization or [reorg]. Reorg depth is bounded by `max_reorg_depth_allowed` (default 10 000 blocks, searched in +500-header chunks); if no chain split is found within that depth the sync peer is banned and no reorg is performed. ### Pruning @@ -160,10 +177,15 @@ block and transaction history is not required. This allows base layer nodes to r Pruning is only for the benefit of the local Base Node, as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon] height. A Base Node will either run in archival mode or pruned mode. -If the Base Node is running in archive mode, it MUST NOT prune. - -When running in pruning mode, [Base Node]s SHOULD remove all spent outputs that are older than the -[pruning horizon] in their current stored [UTXO] set when a new block is received from another [Base Node]. +Mode is determined by `pruning_horizon`: a value of `0` means archival (the shipped default), any non-zero value enables +pruned mode. If the Base Node is running in archive mode (`pruning_horizon == 0`), it MUST NOT prune. + +When running in pruning mode, [Base Node]s SHOULD remove the inputs and spent outputs of every block older than the +[pruning horizon] from their local storage when the chain is extended by a new block from another [Base Node]. Pruning +is gated by `pruning_interval` (default 50 blocks), so it only runs once the local `pruned_height` has fallen more than +`pruning_interval` blocks behind `best_block_height − pruning_horizon`. When a large backlog (more than 25 000 blocks) +must be pruned, the work is performed by a background task in 5 000-block chunks so that normal block processing is not +blocked. Headers and the pruned kernel MMR are always retained. # Change Log diff --git a/src/RFC-0160_BlockSerialization.md b/src/RFC-0160_BlockSerialization.md index b086d2b..6c0821c 100644 --- a/src/RFC-0160_BlockSerialization.md +++ b/src/RFC-0160_BlockSerialization.md @@ -80,41 +80,38 @@ following underlying data types for serialization: 6. `array` of type `[u8; n]` 7. `Vec` where `T` is `u8`, `enum` or `array` -For 1. to 5. and all numbers, [Base 128 Varint] encoding MUST be used. +For 1. to 5., fixed-size little-endian [Borsh] encoding MUST be used: integers are encoded in their minimum-width +little-endian representation (`u8` = 1 byte, `u16` = 2 bytes, `u64`/`i64` = 8 bytes), and `bool` as a single byte +(`0` or `1`). -From the Protocol Buffers documentation: +For 6. to 7., the dynamically sized `array` and `Vec` types MUST be preceded by a 4-byte little-endian length prefix +indicating the number of elements. The elements themselves are then encoded sequentially in declaration order. -> Varints are a method of serializing integers using one or more bytes. Smaller numbers take a smaller number of bytes. -> Each byte in a varint, except the last byte, has the most significant bit (msb) set – this indicates that there are -> further bytes to come. The lower 7 bits of each byte are used to store the two's complement representation of the -> number in groups of 7 bits, least significant group first. +Consensus hashing of these structures uses domain-separated [Borsh] serialization (see [Hash domains](#hash-domains)). -For 6. to 7., the dynamically sized `array` and `Vec` type, the encoded array MUST be preceded by a number indicating the -length of the array. This length MUST also be encoded as a varint. By prepending the length of the array, the decoder -knows how many elements to decode as part of the sequence. - -[base 128 varint]: https://developers.google.com/protocol-buffers/docs/encoding#varints +[borsh]: https://borsh.io/ ## Block field ordering -Using this varint encoding, all fields of the complete block MUST be encoded in the following order: +Using this Borsh encoding, all fields of the complete block MUST be encoded in the following order: 1. Version 2. Height 3. Previous block hash 4. Timestamp -5. Output Merkle root -6. Witness Merkle root -7. Output Merkle mountain range size -8. Kernel Merkle root -9. Kernel Merkle mountain range size -10. Input Merkle root +5. Input Merkle root +6. Output Merkle root +7. Block output Merkle root +8. Output SMT size +9. Kernel Merkle root +10. Kernel Merkle mountain range size 11. Total kernel offset 12. Total script offset -13. Nonce -14. Proof of work algorithm -15. Proof of work supplemental data -16. Transaction inputs - for each input: +13. Validator node Merkle root +14. Validator node Merkle mountain range size +15. Proof of work (algorithm and supplemental data) +16. Nonce +17. Transaction inputs - for each input: - Version - Spent output - for each output: - Version @@ -132,7 +129,7 @@ Using this varint encoding, all fields of the complete block MUST be encoded in - Minimum value promise - Input data ([vector] of Stack items) - Script signature -17. Transaction outputs - for each output: +18. Transaction outputs - for each output: - Version - Features - Version @@ -148,7 +145,7 @@ Using this varint encoding, all fields of the complete block MUST be encoded in - Covenant - Encrypted value - Minimum value promise -18. Transaction kernels - for each kernel: +19. Transaction kernels - for each kernel: - Version - Features - Fee @@ -229,10 +226,9 @@ domain separation. Tari uses the [hashing API](https://github.com/tari-project/t within the tari codebase to achieve proper hash domain separation. The following functional areas MUST each use a separate hash domain that is unique in the tari codebase: -- kernel Merkle Mointain Range; -- witness MMR; -- output MMR; -- input MMR; +- kernel Merkle Mountain Range; +- output sparse Merkle tree (Jellyfish Merkle Tree); +- input Merkle root; - value encryption. To achieve interoperability with other blockchains like Bitcoin and Monero, for example an atomic swap, TariScript MUST @@ -254,16 +250,18 @@ Here we describe the respective Rust types of these fields in the tari codebase, | Height | `u64` | `u64` | Height of this block since the genesis block | | Previous Block Hash | `BlockHash` | `[u8;32]` | Hash of the previous block in the chain | | Timestamp | `EpochTime` | `u64` | Timestamp at which the block was built (number of seconds since Unix epoch) | -| Output Merkle Root | `BlockHash` | `[u8;32]` | Merkle Root of the unspent transaction ouputs | -| Witness Merkle Root | `BlockHash` | `[u8;32]` | MMR root of the witness proofs | -| Output MMR Size | `u64` | `u64` | The size (number of leaves) of the output and range proof MMRs at the time of this header | -| Kernel Merkle Root | `BlockHash` | `[u8;32]` | MMR root of the transaction kernels | -| Kernel MMR Size | `u64` | `u64` | Number of leaves in the kernel MMR | -| Input Merkle Root | `BlockHash` | `[u8;32]` | Merkle Root of the transaction inputs in this block | -| Total Kernel Offset | `BlindingFactor` | `[u8;32]` | Sum of kernel offsets for all transaction kernels in this block | -| Total Script Offset | `BlindingFactor` | `[u8;32]` | Sum of script offsets for all transaction kernels in this block | -| Nonce | `u64` | `u64` | Nonce increment used to mine this block | -| Pow | `ProofOfWork` | See [Proof Of Work](#proof-of-work) | Proof of Work information | +| Input Merkle Root | `BlockHash` | `[u8;32]` | Merkle Root of the transaction inputs in this block | +| Output Merkle Root | `BlockHash` | `[u8;32]` | Sparse Merkle tree (Jellyfish Merkle Tree) root of the unspent transaction outputs | +| Block Output Merkle Root | `BlockHash` | `[u8;32]` | MMR root combining the block's coinbase output hashes and the non-coinbase output MMR root | +| Output SMT Size | `u64` | `u64` | The number of leaves in the output sparse Merkle tree at the time of this header | +| Kernel Merkle Root | `BlockHash` | `[u8;32]` | MMR root of the transaction kernels | +| Kernel MMR Size | `u64` | `u64` | Number of leaves in the kernel MMR | +| Total Kernel Offset | `BlindingFactor` | `[u8;32]` | Sum of kernel offsets for all transaction kernels in this block | +| Total Script Offset | `BlindingFactor` | `[u8;32]` | Sum of script offsets for all transaction kernels in this block | +| Validator Node Merkle Root | `BlockHash` | `[u8;32]` | Merkle root of all active validator nodes | +| Validator Node Size | `u64` | `u64` | Number of validator node hashes | +| Pow | `ProofOfWork` | See [Proof Of Work](#proof-of-work) | Proof of Work information | +| Nonce | `u64` | `u64` | Nonce increment used to mine this block | `[u8;32]` indicates an array of 32 unsigned 8-bit integers @@ -271,7 +269,7 @@ Here we describe the respective Rust types of these fields in the tari codebase, | Field | Abstract Type | Data Type | Description | |-------------------------|------------------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Proof of Work Algorithm | `PowAlgorithm` | `u8` | The algorithm used to mine this block ((Monero or SHA3)) | +| Proof of Work Algorithm | `PowAlgorithm` | `u8` | The algorithm used to mine this block (0 = RandomX-M, 1 = Sha3x, 2 = RandomX-T, 3 = Cuckaroo) | | Proof of Work Data | `Vec` | `u8` | Supplemental proof of work data. For example for Sha3, this would be empty (only the block header is required), but for Monero merge mining we need the Monero block header and RandomX seed hash | ### Block Body diff --git a/src/RFC-0170_NetworkCommunicationProtocol.md b/src/RFC-0170_NetworkCommunicationProtocol.md index b9c7147..0f22a96 100644 --- a/src/RFC-0170_NetworkCommunicationProtocol.md +++ b/src/RFC-0170_NetworkCommunicationProtocol.md @@ -126,11 +126,11 @@ In the Tari communication network, each CN or CC makes use of a node ID to deter This node ID can be derived from the CNs or CCs identification public key. The method used to obtain a node ID will either enhance or limit the trustworthiness of that entity when propagating messages through them on the Tari communication network. -The similarity or distance between different node IDs can be calculated by performing the [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between the bits of the two node ID numbers. -The Hamming distance can be implemented as an Exclusive OR (XOR) between the bits of the numbers and the summation of the resulting true bits. -CCs and/or CNs that have similar node IDs, that produce a small Hamming distance, are located in similar regions of the Tari communication network. +The similarity or distance between different node IDs is calculated using the XOR distance metric as proposed by the [Kademlia](https://en.wikipedia.org/wiki/Kademlia) paper. +The distance is computed by taking the Exclusive OR (XOR) of the two node IDs and interpreting the result as an unsigned integer for comparison (not by summing the differing bits, which would be the Hamming distance). +CCs and/or CNs that have similar node IDs, that produce a small XOR distance, are located in similar regions of the Tari communication network. This does not mean that their geographic locations are near each other, but rather that their location in the network is similar. -A thresholding scheme can be applied to the Hamming distance to ensure that only neighboring CNs with similar node IDs are allowed to share and propagate specific information. +A thresholding scheme can be applied to the XOR distance to ensure that only neighboring CNs with similar node IDs are allowed to share and propagate specific information. As an example, only routing table information that contains similar node IDs to the requesting CCs or CNs node ID should be shared with them. Limiting the sharing of routing table information makes it more difficult to map the entire Tari communication network. @@ -142,15 +142,7 @@ The parent Base Node will perform any communication tasks on the Tari communicat #### Online Communication Address, Peer Address and Routing Table Each CC and CN on the Tari communication network will have identification cryptographic keys, a node ID and an online communication address. -The online communication address SHOULD be either an IPv4, IPv6, Or Tor (Base32) address and can be stored using the network address type as follows: - -| Description | Data type | Comments | -|:-------------|:-----------|:----------------------------------------------------| -| address type | uint4 | Specify if IPv4/IPv6/Tor| -| address | char array | IPv4, IPv6, Tor (Base32) address | -| port | uint16 | port number | - -Tari uses the [Multiaddr] format for addresses. +The online communication address is encoded using the [Multiaddr] format, which self-describes the protocol, address and port. Supported transports are IPv4, IPv6, Onion (Tor), and Memory (in-process); for example `/ip4/123.123.123.123/tcp/12345` or `/onion3/abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrst:12345`. A Tor address can be used when anonymity is important for a CC or CN. The IPv4 and IPv6 address types do not provide any privacy features but do provide increased bandwidth. @@ -261,7 +253,7 @@ The CN can decide how the peer connections should be selected from the routing t - A CN MUST construct and provide a list of peer addresses from its routing table that is similar to a requested node ID so that other CCs and CNs can extend their routing tables. - A CN MUST keep its routing table up to date by removing unreachable peer addresses and adding newly received addresses. - It MUST have a mechanism to determine if a node ID was obtained through registration or was derived from an identification public key. -- A CN MUST calculate the similarity between different node IDs by calculating the Hamming distance between the bits of the two node ID numbers. +- A CN MUST calculate the similarity between different node IDs by calculating the XOR distance between the two node ID numbers. #### Functionality Required of Communication Clients @@ -274,7 +266,7 @@ The CN can decide how the peer connections should be selected from the routing t - A CC MUST maintain a small persistent routing table of Tari Communication network peers with which ad hoc connections can be established. - As the CC becomes aware of other CNs and CCs on the communication network, the CC SHOULD extend its local routing table by including the newly discovered CCs or CNs contact information. - Peers from the CCs routing table that have been unreachable for a number of attempts SHOULD be removed from the its routing table. -- A CC MUST calculate the similarity between different node IDs by calculating the Hamming distance between the bits of the two node ID numbers. +- A CC MUST calculate the similarity between different node IDs by calculating the XOR distance between the two node ID numbers. # Change Log diff --git a/src/RFC-0172_PeerToPeerMessagingProtocol.md b/src/RFC-0172_PeerToPeerMessagingProtocol.md index 403466f..35c1244 100644 --- a/src/RFC-0172_PeerToPeerMessagingProtocol.md +++ b/src/RFC-0172_PeerToPeerMessagingProtocol.md @@ -127,6 +127,9 @@ The following transports are supported by the Tari communication layer: - TCP/IP - A publicly accessible IPv4 address. - SOCKS5 - Allows a SOCKS5 proxy to be configured for inbound and outbound connections. - Tor - A specialisation of the SOCKS5 transport that facilitates connections over the Tor network. +- TorTcp - Enables both Tor and TCP peer addresses, preferring Tor when selecting dial addresses. +- TcpTor - Enables both TCP and Tor peer addresses, preferring TCP when selecting dial addresses. This is the default transport. +- Memory - In-process transport used for testing. #### Establishing a Connection @@ -165,8 +168,9 @@ We list the following characteristics and requirements for encrypted peer-to-pee - forward secrecy; and, - for efficiency, has a minimal round trip time. -For these reasons we select the single round-trip `Noise_IX_25519_ChaChaPoly_BLAKE2b` protocol, that is, the Noise IX +For these reasons we select the `Noise_XX_25519_ChaChaPoly_BLAKE2b` protocol, that is, the Noise XX handshake pattern using Curve25519 for ephemeral and static identities, ChaChaPoly encryption and a HKDF using BLAKE2b. +A prologue `com.tari.comms.noise.prologue` is set on both sides to bind the handshake to the Tari protocol. If successful, an authenticated encrypted socket connection is established between the peers. @@ -205,8 +209,9 @@ To begin, the initiator MUST send a protocol negotiation message consisting of 1 The bitflags are defined as follows: - `0x01` - `OPTIMISTIC` - `0x02` - `TERMINATE` -- `0x04` - `PROTOCOL_NOT_SUPPORTED` (response) -- `0x08 - 0x128` - Future use (ignored) +- `0x04` - `NOT_SUPPORTED` (response) + +When the `OPTIMISTIC` flag is set, the `TERMINATE` flag is also set so that the initiator does not wait for a response. If the `OPTIMISTIC` flag is set, the initiator considers the negotiation complete as it is optimistic that the responder supports it. It can assume this because the peer gave a list of supported protocols in the [Identity Exchange](#identity-exchange) @@ -218,7 +223,7 @@ flag unset in the initial message. If the responder supports the protocol, it SHOULD respond with the name of the supported protocol and all flags unset and immediately proceed with the agreed upon protocol. -If not, it SHOULD respond with the `PROTOCOL_NOT_SUPPORTED` flag set and an empty protocol name and wait for more messages. +If not, it SHOULD respond with the `NOT_SUPPORTED` flag set and an empty protocol name and wait for more messages. The initiator MAY send another protocol negotiation message or close the substream. A responder MAY set the `TERMINATE` flag at any time and close the substream. In practise, this is used to indicate to the @@ -235,7 +240,7 @@ Fields include: as preferred address; - `peer_features` - bitflags with the following flags - `MESSAGE_PROPAGATION = 0x01` - peer is able to propagate/route messages - - `DHT_STORE_FORWARD = 0x02` - peer provides message storage and can respond to `SafRequestMessages` + - `DHT_STORE_FORWARD = 0x02` - legacy flag for nodes that used to offer store and forward functionality. This is no longer used but is retained for backward compatibility (its presence affects the node's public key derivation). Store-and-forward messages are not supported and are discarded. - A `COMMUNICATION_NODE` is defined as `0x03` (`MESSAGE_PROPAGATION | DHT_STORE_FORWARD`) - A `COMMUNICATION_CLIENT` is defined as `0x00` - `last_seen` - a timestamp of the last time a message has been sent/received from this peer; @@ -331,8 +336,8 @@ The `DhtHeader` is a protobuf message with these fields: - Join = 1 - Join/Announce - Discovery = 2 - Discovery request - DiscoveryResponse = 3 - Response to a discovery request - - SafRequestMessages = 20 - Request stored messages from a node - - SafStoredMessages = 21 - Stored messages response + - SafRequestMessages = 20 - (Deprecated/removed) Request stored messages from a node. These messages are discarded by nodes that do not support store-and-forward (all current nodes). + - SafStoredMessages = 21 - (Deprecated/removed) Stored messages response. - `flags: u32` - bitflags `0x01 - ENCRYPTED` - `message_tag: u64` - Message trace ID. This can be omitted or any value and is used for debug tracing. @@ -367,14 +372,13 @@ If any of these rules fail the message SHOULD be discarded. The protocol provides for the following outbound message broadcast strategies: -- `Direct(Identity)` - Send the message directly to the destination peer. +- `DirectNodeId(NodeId)` - Send the message directly to the peer matching the given node ID. +- `DirectPublicKey(CommsPublicKey)` - Send the message directly to the peer matching the given public key. - `Flood(exclude)` - Send to all connected peers excluding `exclude` peers. If no peers are connected, no messages are sent. - `Random(n, exclude)` - Send to a random set of peers of size n that are Communication Nodes, excluding `exclude` peers. -- `ClosestNodes({node_id, exclude, connected_only})` - Send to all n nearest Communication Nodes to the given node_id. -- `DirectOrClosestNodes({node_id, exclude, connected_only})` - Send directly to destination if connected but otherwise send to all n nearest Communication Nodes -- `Broadcast(excludes)`- Send to a random set of _connected_ peers, excluding `excludes` peers. The number of peers selected at most equal to `propagation_factor`. +- `Broadcast(excludes)`- Send to a random set of _connected_ peers, excluding `excludes` peers. The number of peers selected is at most equal to `broadcast_factor`. - `SelectedPeers(peers)` - Send to the specified peers. -- `Propagate(NodeDestination, Vec)` - Propagate to a set of _connected_ peers closer to the destination. The number of peers selected at most equal to `propagation_factor`. +- `Propagate(NodeDestination, Vec)` - Propagate to a set of _connected_ peers closer to the destination. The number of peers selected is at most equal to `propagation_factor`. A peer's node_id is defined as the 13-byte variable-length Blake2b hash of the public key. To determine if a peer identity is "closer" to another peer we compare the XOR distance between peers as proposed by the [kademlia] paper. @@ -412,14 +416,15 @@ Every Tari message MUST have a payload header containing the following fields at | Name | Type | Description | |--------------|-------|-------------------------------------------------------------------------------| -| message_type | `u8` | An enumeration of the message type of the body. Refer to message types below. | -| nonce | `u32` | The optional message nonce. | +| message_type | `int32` | An enumeration of the message type of the body. Refer to message types below. | +| nonce | `uint64` | The optional message nonce. | MessageTypes are represented as an unsigned eight-bit integer denoting the expected contents of the `MessageBody`. | Category | Name | Value | Description | |------------|:--------------------------------|-------|---------------------------------------------------------------------------------------------| -| Network | PingPong | 1 | A PongPong message. | +| Network | PingPong | 1 | A PingPong message. | +| Network | Chat | 2 | A chat message. | | Blockchain | NewTransaction | 65 | Transaction submitted by a wallet or propagated by a base node. | | Blockchain | NewBlock | 66 | Block propagated by a base node. | | Wallet | SenderPartialTransaction | 67 | A partial MimbleWimble transaction submitted by a sender wallet to the receiver. | @@ -430,6 +435,8 @@ MessageTypes are represented as an unsigned eight-bit integer denoting the expec | Blockchain | MempoolResponse | 72 | Base node response in reply to a MempoolRequest message. | | Wallet | TransactionFinalized | 73 | Finalized transaction message sent by a sender to receiver wallet. | | Wallet | TransactionCancelled | 74 | A courtesy message sent by a wallet to inform the other that the transaction is cancelled. | +| Extended | Text | 225 | A text message. | +| Extended | TextAck | 226 | A text message acknowledgement. | All other message types are reserved for future use. @@ -447,7 +454,7 @@ To route an encrypted message, the following requirements MUST be met: A message is encrypted using the following procedure: - The `DhtEnvelopBody` is containing the `MessageHeader` and `MessageBody` is serialized using [protobuf]. -- A CSRNG is used to generate the cipher nonce and this is prepended onto the message. +- A fixed zero nonce is used for the AEAD cipher. This is safe because the message key is ephemeral (derived from a fresh ECDH exchange per message). - The plaintext message is padded with '0x00' to a multiple of 6000 bytes. - The message encryption key is generated as follows: - Key material `dh_key` is generated by Diffie-Hellman of the recipient public key and the ephemeral private key. diff --git a/src/RFC-0173_Versioning.md b/src/RFC-0173_Versioning.md index 68efe94..c3a3bba 100644 --- a/src/RFC-0173_Versioning.md +++ b/src/RFC-0173_Versioning.md @@ -96,21 +96,24 @@ connection cheaply without having to perform any further operations, like comple The following is the current mapping of the WireMode byte: ```rust,ignore - pub enum Network { +pub enum Network { MainNet = 0x00, + StageNet = 0x01, + NextNet = 0x02, LocalNet = 0x10, - Ridcully = 0x21, - Stibbons = 0x22, - Weatherwax = 0xa3, Igor = 0x24, - Dibbler = 0x25, Esmeralda = 0x26, } // As well as the special wiremode for local liveness checks -const LIVENESS_WIRE_MODE: u8 = 0xa6; +const LIVENESS_WIRE_MODE: u8 = 0xa7; ``` +Note: The wire byte sent on the wire is not always equal to the enum discriminant. `Network::as_wire_byte()` returns a +value within a binned range for each network (e.g. `NextNet` has discriminant `0x02` but wire byte `82`; `Esmeralda` +has discriminant `0x26` but wire byte `202`). The liveness wire byte `0xa7` is reserved and must never be used for a +network. + ### P2P message version Peer to Peer messages on the Tari network are encapsulated into message envelopes. The body of message envelopes are defined, serialized and deserialized using Protobuf. These messages will only be updated by adding new fields to the diff --git a/src/RFC-0174_Chat_Metadata.md b/src/RFC-0174_Chat_Metadata.md index 19c8f61..837b447 100644 --- a/src/RFC-0174_Chat_Metadata.md +++ b/src/RFC-0174_Chat_Metadata.md @@ -5,7 +5,7 @@ | Title | Chat Metadata | | Last Modified | 2026-07-16 | | Authors | Tari Labs | -| Status | Depricated | +| Status | Deprecated | | Type | RFC | | Created | 2023-12-21 | | References | | @@ -67,6 +67,8 @@ will use during interaction with other nodes. ## Description +**Note:** This RFC describes a proposed chat metadata protocol that was never implemented in the Tari codebase. The message type values `TariMessageTypeChat = 2`, `TariMessageTypeText = 225`, and `TariMessageTypeTextAck = 226` are reserved in the protocol enum, but no chat metadata structures or handlers exist in the v5.5.0 release. + The chat protocol incorporates metadata within messages to support various message types, such as Replies, TokenRequests, Gifs, and Links. This metadata-driven approach enhances the technical flexibility of chat, allowing for the structured handling of distinct content and interactions within the messaging framework. diff --git a/src/RFC-0182_CommitmentSignatures.md b/src/RFC-0182_CommitmentSignatures.md index 441bb1a..a9d2325 100644 --- a/src/RFC-0182_CommitmentSignatures.md +++ b/src/RFC-0182_CommitmentSignatures.md @@ -12,7 +12,7 @@ ## Commitment and public key signatures -![status: draft](theme/images/status-draft.svg) +![status: stable](theme/images/status-stable.svg) # Licence @@ -61,7 +61,7 @@ This Request for Comment (RFC) describes signatures relating to commitments and ## Related Requests for Comment -* [RFC-0120: TariScript](./RFC-0120_TariScript.md) +* [RFC-0201: TariScript](./RFC-0201_TariScript.md) ## Introduction diff --git a/src/RFC-0190_Mempool.md b/src/RFC-0190_Mempool.md index 9942949..48d58c1 100644 --- a/src/RFC-0190_Mempool.md +++ b/src/RFC-0190_Mempool.md @@ -112,32 +112,33 @@ can be configured. When a new transaction is received and the storage capacity l transactions are prioritized by ordering their total `fee per gram` of all UTXOs used and transaction age, in that order. The transactions of the least priority are discarded to make room for higher priority transactions. -The transaction priority metric has the following behavior: -- transactions with higher fee per gram **SHOULD** be prioritized over lower fee per gram transactions. -- older transactions in the mempool **SHOULD** be prioritized over newer ones. +The transaction priority metric is a composite key with the following ordering (highest to lowest significance): +- **Fee per gram** (scaled by 1000 for integer precision): `total_fee * 1000 / weight`. Transactions with higher fee per gram **SHOULD** be prioritized over lower fee per gram transactions. +- **Transaction age** (`u64::MAX - insert_epoch`): older transactions in the mempool **SHOULD** be prioritized over newer ones. +- **Aggregate excess signature and nonce**: used as a tiebreaker to ensure unique keys in the priority index. ### Memory Pool State: Syncing and Updating -On the initial startup, the complete state of the unconfirmed pool is pulled from the connected peers. Typically, Mempool -doesn't persist its state, but can be configured to do so. If the state is locally present, then only the missing, unconfirmed transactions -are synced from the peers, otherwise, the full state is requested. The validity and priority of transactions are computed -as they are being downloaded from the connected peers. If the base node undergoes a re-org, then the missing state is again -pulled from the peers. +On the initial startup, after the node has completed chain sync, the mempool state is synchronized with up to +`initial_sync_num_peers` (default: 2) connected peers using a bidirectional inventory-based protocol. Each peer sends a +transaction inventory (list of kernel excess signatures); the receiving peer streams back transactions not already known, +then requests any missing transactions from the sender. Each sync session is capped at `initial_sync_max_transactions` +(default: 10,000) transactions. The Mempool does not persist its state; on restart the pool is empty and the full state is +synced from peers. The validity and priority of transactions are computed as they are received from the connected peers. +If the base node undergoes a re-org that adds at least `block_sync_trigger` (default: 5) blocks, a mempool sync is +re-triggered with peers. The functional bahavior required for the Mempool's synchronization are the following: -- All verified transactions **MUST** be propagated to neighboring peers. +- All verified transactions that are stored in the Unconfirmed Pool **MUST** be propagated to neighboring peers. Transactions that fail validation (orphan, time-locked, already spent, duplicate kernel, fee too low, consensus failure) **MUST NOT** be propagated. - Unverified or invalid transactions **MUST NOT** be propagated to peers. - Verified transactions that were discarded due to low priority levels **MUST** be propagated to peers. - Duplicate transactions **MUST NOT** be propagated to peers. - Mempool **MUST** have an interface, allowing peers to query and download its state, partially and in full. - Mempool **MUST** accept all transactions received from peers but **MAY** decide to discard low-priority transactions. - Mempool **MUST** allow wallets to track payments by monitoring that a particular transaction has been added to the Mempool. -- Mempool **MAY** choose: - - to discard its state on restart and then download the full state from its peers or - - to store its state using persistent storage to reduce communication bandwidth required when - reinitializing the pool after a restart. +- The Mempool discards its state on restart and downloads the full state from its peers. Persistent storage of the mempool state is not supported. ### Unconfirmed Pool @@ -151,7 +152,7 @@ Functional behavior required of the Unconfirmed Pool: - It **MUST** ensure that incoming transactions don't have a processing time-lock or has a time-lock that has expired. - It **MUST** ensure that all time-locks of the UTXOs that will be spent by the transaction have expired. -- Transactions that have been used to construct new blocks **MUST** be removed from the Unconfirmed Pool and added to the Reorg Pool. +- Transactions that have been used to construct new blocks **MUST** be removed from the Unconfirmed Pool and added to the Reorg Pool. Transactions that conflict with a published block (double-spending inputs already spent by the block, or producing outputs already produced by the block) **MUST** be removed from the Unconfirmed Pool and discarded. ### Reorg Pool @@ -159,13 +160,16 @@ The Reorg Pool consists of transactions that have recently been added to blocks, their removal from the Unconfirmed Pool. When a potential blockchain reorganization occurs that invalidates previously assembled blocks, the transactions used to construct these discarded blocks can be moved back into the Unconfirmed Pool. This ensures that high-priority transactions are not lost during reorganization and can be included in future blocks. -The Reorg Pool is an internal, isolated Mempool component and cannot be accessed or queried from outside. +The Reorg Pool is an internal Mempool component that is not directly accessible, but its state can be queried through the Mempool's public interface (e.g., via `get_state`, `get_tx_state_by_excess_sig`, and `retrieve_by_excess_sigs`). Functional behavior required of the Reorg Pool: - Copies of the transactions used recently in blocks **MUST** be stored in the Reorg Pool. -- Transactions in the Reorg Pool **MAY** be discarded after a set expiration threshold has been reached. +- Transactions in the Reorg Pool **MAY** be discarded after a set block-height expiration horizon (`expiry_height`, default: 5 blocks) has been reached. - When reorganization is detected, all affected transactions found in the Reorg Pool **MUST** be moved back to the - Unconfirmed Pool and removed from the Reorg Pool. + Unconfirmed Pool and removed from the Reorg Pool. Additionally, all transactions currently in the Unconfirmed Pool + **MUST** be drained and re-validated, as previously valid transactions may no longer be valid after the reorg. + Double-spending transactions in the Reorg Pool (whose inputs are spent by new blocks in the reorg) **MUST** be + discarded. ### Mempool @@ -174,15 +178,14 @@ must pass all verification steps to make it into the Unconfirmed Pool and furthe Functional behavior required of the Mempool: - If the received transaction already exists in the Mempool, then it **MUST** be discarded. -- If multiple transactions contain the same UTXO, then only the highest priority transaction **MUST** be kept and the - rest (having the said UTXO included) **MUST** be discarded. +- If multiple transactions contain the same UTXO, they **MAY** coexist in the Unconfirmed Pool. When constructing a new block, only one transaction per conflicting input is selected (the highest priority one that fits); the rest remain in the pool for potential inclusion in future blocks. - If the storage capacity limit is reached, then new incoming transactions **SHOULD** be prioritized according to a set number of [rules](#prioritizing-unconfirmed-transactions). - Transactions of the least priority **MUST** be discarded to make room for higher priority incoming transactions. -- Transactions with its computed priority being lower than the minimum set threshold **MUST** be discarded. +- Transactions with a total fee lower than the configured `min_fee` threshold (default: 50 MicroMinotari) **MUST** be discarded. Additionally, when the Unconfirmed Pool is at capacity, incoming transactions with a priority lower than the current lowest-priority transaction in the pool **MUST** be discarded. - The Mempool **MUST** verify that incoming transactions do not have duplicate outputs. - The Mempool **MUST** verify that only matured coinbase outputs can be spent. -- Each component pool **MAY** have the storage capacity configured and adjusted. +- The Unconfirmed Pool **MAY** have its storage capacity (maximum number of transactions, default: 40,000) configured and adjusted. The Reorg Pool **MAY** have its expiry height horizon (default: 5 blocks) configured and adjusted. - The memory pool **SHOULD** have a mechanism to estimate fee categories from the current Mempool state. For example, the transaction fee can be estimated, ensuring that new transactions will be properly prioritized, to be added into new blocks in a timely manner. @@ -191,7 +194,7 @@ Functional behavior required for the allocation of incoming transactions in the - Verified transactions that have passed all checks such as spending of valid UTXOs and expired time-locks **MUST** be placed in the Unconfirmed Pool. - Incoming transactions with time-locks, prohibiting them from being included in new blocks **SHOULD** be discarded. -- Newly received, verified transactions attempting to spend a UTXO that does not yet exist **MUST** be discarded. +- Newly received, verified transactions attempting to spend a UTXO that does not yet exist in the chain **MUST** be discarded, unless the UTXO is an output of another transaction already in the Unconfirmed Pool, in which case the transaction is stored as a dependent transaction with a reference to its dependent outputs. - Transactions that have been added to blocks and were removed from the Unconfirmed Pool **SHOULD** be added to the Reorg Pool. [base layer]: Glossary.md#base-layer diff --git a/src/RFC-0202_TariScriptOpcodes.md b/src/RFC-0202_TariScriptOpcodes.md index 4448f97..8becc25 100644 --- a/src/RFC-0202_TariScriptOpcodes.md +++ b/src/RFC-0202_TariScriptOpcodes.md @@ -114,7 +114,7 @@ The full list of [Error codes](#error-codes) is given below. ### Constraints -* The maximum length of a script when serialised is 1,024 bytes. +* The maximum length of a script when serialised is 512 bytes (consensus-enforced; the TariScript library hard-caps parsing at 4,096 bytes). * The maximum length of a script's input is 1,024 bytes. * The maximum stack height is 255. diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ee06e9c..03d606a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -45,7 +45,7 @@ - [MT-0230: Hash time locked contracts](RFC-0230_HTLC.md) - [MT-0240: Atomic swap](RFC-0240_AtomicSwap.md) - [MT-8001: Multi-party transactions](RFC-8001_MultiPartyTransactions.md) -- [Depricated](deprecated.md) +- [Deprecated](deprecated.md) - [RFC](deprecated_rfc.md) - [MT-0010: Tari code structure and organization](RFCD-0010_CodeStructure.md) - [MT-0121: Consensus encoding](RFCD-0121_ConsensusEncoding.md) diff --git a/src/base_layer.md b/src/base_layer.md index 019efea..09e3eb0 100644 --- a/src/base_layer.md +++ b/src/base_layer.md @@ -10,16 +10,17 @@ The Tari Base Layer network comprises the following major pieces of software: [Noise protocol](https://noiseprotocol.org/) and [Tor](https://www.torproject.org/) to be highly secure and anonymous. Devices behind NATs and firewalls can use Tari's communication tools with ease. - Mining software. Miners perform proof-of-work to secure the base layer and compete to submit the - next valid block into the Tari blockchain. Tari uses two Proof of Work (PoW) algorithms, the first is - [merge-mined](RFC-0132_Merge_Mining_Monero.md) - with Monero, and the second is the native [SHA3x](RFC-0131_Mining.md) algorithm. - The Tari source provides three alternatives for Tari miners: - - A standalone miner for SHA3 mining - - A merge-mining proxy to be used with XMRig to merge mine Tari with Monero + next valid block into the Tari blockchain. Tari uses a hybrid of four Proof of Work (PoW) algorithms: + [merge-mined](RFC-0132_Merge_Mining_Monero.md) Monero RandomX (`RandomXM`), the native [SHA3x](RFC-0131_Mining.md) + algorithm, solo Tari RandomX (`RandomXT`), and Cuckaroo-29 (`C29`). Each algorithm has an independent difficulty, + and the four combined target an average block interval of 120 seconds. + The Tari source provides these miners for Tari: + - A standalone miner (`minotari_miner`) capable of SHA3x and RandomXT mining + - A merge-mining proxy (`minotari_merge_mining_proxy`) to be used with XMRig to merge mine Tari with Monero - Wallet software. Client software and Application Programming Interfaces (APIs) offering means to construct transactions, query nodes for information and maintain personal private keys. The reference design includes a - wallet library, an C FFI interface, gRPC client and server code, a multi-platform console text-based wallet, and - Aurora, the mobile wallet. + wallet library, a C FFI interface, gRPC client and server code, a multi-platform console text-based wallet, and + Aurora, the mobile wallet. The RFCs in this section go into great describing how the various components work, and how they fit together to provide the backbone of the Tari ecosystem.