Malachite v0.3.0 testing regimen

[adizere]

This is a comprehensive announcement prompted by the following question:

What kind of testing has been done with Malachite, as of v0.3.0 https://github.com/informalsystems/malachite/releases/tag/v0.3.0 ?

Testing regimen so far

Periodic performance benchmarks

We publish regular performance benchmarks on the dashboard. These are done using the Starknet testing sequencer we wrote and this QA infrastructure.

Briefly, these tests help to catch regressions and to give a sense of the engine's performance using the Starknet sequencer, and its history over time.

Model-based testing

Parts of the codebase were co-designed with their formal specification and model checking. These parts specifically correspond to the Tendermint algorithm, i.e., the per-round state machine and the VoteKeeper.

To see the complete specs: https://github.com/informalsystems/malachite/tree/v0.3.0/specs , and the MBT code is in: https://github.com/informalsystems/malachite/tree/v0.3.0/code/crates/test/mbt

Unit and integration tests

The core components of Malachite are also tested extensively with unit tests. Those components are:

Per-round state machine (96% coverage at this time)
Vote keeper (94% coverage at this time)
Driver (90% coverage at this time)

Moreover, the whole Malachite engine is tested thoroughly using end-to-end integration tests. These tests are written in a custom DSL (domain-specific language) which allows to:
Configure and spawn multiple nodes using our production libp2p-based networking layer,
Crash and restart nodes at specific points, eg. crash a node after it has just proposed a value, restart a node after the rest of the network has reached a given height or round
Override the behavior of a node on-demand: e.g. by forcing a node to vote nil on every proposal, or to propose two different values within the same height, etc.
Set and check expectations on the behaviour of these nodes: e.g. check that a node requested sync from another node or that it re-proposed the same value twice in different rounds, etc.

https://github.com/informalsystems/malachite/blob/main/CONTRIBUTING_CODE.md#unit-and-integration-tests

Internal protocol review

In Spring 2025, we conducted a comprehensive review of Malachite’s liveness mechanism. While Malachite implements the Tendermint algorithm, which relies on the gossip property for liveness, it does not implement this property directly. Instead, it includes sub-protocols—such as vote rebroadcast and vote set sync—intended to approximate it. Our analysis showed that these sub-protocols were not sufficient to guarantee liveness. To understand the gaps, we analyzed Tendermint’s liveness proof in detail and identified the precise points where liveness depends on the gossip property.

Based on this analysis, we identified a set of concrete mechanisms that can replace the need for the gossip property. A RoundSync mechanism ensures that all correct nodes progress through rounds even in the presence of Byzantine behavior, message loss, or asynchrony, while PolkaReproposal and PolkaHiddenLock ensure that eventually, in a round with a correct proposer, a proposed value will be accepted by all correct nodes and decided.

With these mechanisms in place, Malachite no longer relies on the gossip property—it only requires a simple broadcast primitive that ensures a replica can send messages to all other nodes, independent of the underlying implementation, which can be gossip-based or not. We intend to publish a more extensive blog post on the topic of the gossip property requirements in Tendermint and how we’ve tackled it in Malachite.

Large scale performance experiments

Earliest experiments, October 2024

Our first large-scale experiments with Malachite were conducted in October 2024 and reported in this blog post. We deployed networks with up to 100 validators and used 1MB blocks, observing block times between 700 ms and 860 ms (average 780 ms). This resulted in throughput of approximately 1,270 transactions per second (TPS), or about 1.3 MB/s, assuming 1KB transactions.
We also ran experiments with smaller validator sets. In a 20-node setup, we measured an average finalization latency of 385 ms and achieved throughput of approximately 2,440 TPS (≈2.5 MB/s). To test scalability in this configuration, we increased the block size to 10MB and 50MB. At 10MB, the average latency was 745 ms, with throughput reaching around 13,180 TPS (≈13.5 MB/s). With 50MB blocks, the protocol maintained stability, finalizing blocks in an average of 4.4 seconds and achieving about 10,740 TPS (≈11 MB/s).

Automated large-scale experiments, since May 2025

We recently integrated and automated our QA infrastructure to run regularly through a GitHub workflow, with the goal of executing it for every major release. The workflow currently supports two deployment setups:
A North America setup with 4 validators distributed across New York (two separate datacenters), Toronto, and San Francisco.

A Global setup with 20 validators across 10 regions worldwide, with 2 validators per region. The regions include: New York (two datacenters), Amsterdam, Frankfurt, London, San Francisco, Toronto, Singapore, Sydney, and Bangalore.

In our latest experiments using the v0.3.0 release and 1MB blocks, we measured block times of approximately 100 ms in the North America setup and 330 ms in the Global setup. This resulted in throughput exceeding 10,000 TPS (≈10.2 MB/s) in North America and over 3,000 TPS (≈3.1 MB/s) globally.

Interoperability tests

We conducted interoperability tests between the Starknet sequencer and the Malachite Starknet app. For that, we developed and automated and scalable local setup as described here. We wanted to ensure that consensus can progress seamlessly with non-empty blocks. We observed an issue with the consensus being slow down which was fixed afterwards.
Currently, there is no interoperability issue between the sequencer and Malachite. (context: https://github.com/informalsystems/malachite/pull/868#issuecomment-2901429427)

Networking fitness tests

During Q3 2024 - Q1 2025, we conducted extensive performance testing for the various networking layers of Malachite. More specifically, this included tests across three domains of networking:

A new protocol for node discovery, and
for the Dynamic Optimal Graph (DOG) protocol, an alternative to GossipSub

We use libp2p as a basis for the networking layer. The library provides solid primitives for implementing the exploration blocks above.

In (1), we developed a custom, modular, scalable discovery protocol. The tests were motivated by the need to design a new protocol (adaptable to any network type) that could be bundled with the Malachite engine. The protocol and its implementation became the discovery crate, which we maintain as part of the engine. Moreover, please find a proper evaluation of the protocol in Chapter 3 of the Master’s thesis of Bastien Faivre (EPFL, Distributed Computing Laboratory, 2025).

We’ve used some of the experience & results from testing domains in (1) to inform libp2p and GossipSub configuration parameters for subsequent performance experiments with Malachite and a Starknet sequencer -- see “Large scale performance experiments”. Concretely, these tests led to validation of our choice of libp2p and GossipSub as a default p2p option for Malachite engine, as well as fine-tuning various configuration parameters for those libraries to make them ready for use by applications on top of Malachite.

Our investigations and tests into domain (2) were inspired by the promising results of the DOG protocol as part of the CometBFT codebase. We’ve designed and implemented DOG on top of libp2p (at the same level as GossipSub) and then tested it as part of an MVP in libp2p-dog. Please refer to Chapter 4 of the Master’s thesis of Bastien Faivre (EPFL, Distributed Computing Laboratory, 2025) for a more comprehensive description of the benchmarks between GossipSub and DOG.

Note that both tests were executed on a realistic worldwide setup. More information on the respective descriptions.

Snapchain adversarial tests

Since mid-May 2025, we have been running a series of adversarial testing targeted at the Snapchain application. These tests do not target Malachite narrowly, but instead the Snapchain application built with the Malachite engine. We manually perturb the network with e.g. high latency, bandwidth limitations, stampedes (many nodes quitting or joining), validator crashes and restarts, and misconfiguring the network stack towards uncovering typical problems.

These tests have surfaced issues with block sync, which was diagnosed as being too slow in certain conditions, and it led to performance improvements to the sync sub-system (parallelization of sync requests, which was implemented in #1095 and will be released after 0.3.0; batching of sync requests, which is in progress) as well as a peer scoring mechanism (implemented in #1071 and released in v0.3.0).

Ongoing and plans before releasing a v1

For transparency, here are the various initiatives we’ve either started, planned, or scoped to do before tagging a v1 for the Malachite engine. Ordered in decreasing order of certainty and priority:

Integration with Apollo (a Starknet sequencer that StarkWare built) & Juno (a Starknet sequencer that Nethermind built) as a follow-up and next milestone in the “Interop testing” domain. Like ongoing interop tests, the goals are to validate that a heterogeneous network (a type of N-version programming) of Starknet sequencers including those based on Malachite can interoperate correctly and to uncover bugs in the various sequencer implementations. The two Sequencer implementations based on Malachite’s consensus libraries are Madara & Pathfinder.

Long-running testnet with a EVM-based application on top of Malachite: We are developing a complete EVM-based application stack that is production ready using Malachite and reth. The validation method for that will be an internal devnet and then a long-running public-facing testnet.

Deterministic simulation: We are currently investigating building a deterministic simulation harness for Malachite, inspired by the relevant work in TigerBeetle, which would allow us to simulate time, network latency, packet loss, partitions, and more, purely in software and to run thousands of more comprehensive tests in the blink of an eye.

Call to action

Finally, we’re soliciting input from auditors and (potential) users on: (1) whether there is value in a conventional security audit of core libraries of Malachite and (2) what would be the exact scope of such an audit?

From our experience with CometBFT, an analysis à la Jepsen might prove more meaningful in uncovering problems with consensus engines (compared to a conventional audit). We are therefore open to ideas and suggestions!

Reach out to us on the MalachiteEngine Telegram group and keep up to date with news on the official Malachite website.