Mainsail's Performance Improvements

In the most recent update, we’ve made significant enhancements to Mainsail, boosting both its performance and resilience. This blog post will delve into these improvements and explore their positive effects on system performance.

#Worker Processes

Child processes were replaced with worker threads. Worker threads are lighter and require fewer resources compared to child processes because additional Node.js instances do not need to be started. This improves thread boot time and reduces communication overhead between threads.

Previously, only the crypto worker ran inside a child process to improve performance by moving serialization, deserialization, and signature verification off the main thread. After the changes, exposed APIs run inside a worker process to keep the main process with consensus stable.

The following diagram shows how APIs, databases, and processes are used:

#Transaction Pool

The transaction pool service is no longer part of the main process. Instead, all its logic has been moved to the transaction-pool-service package. Below, we’ll cover how different parts of it work.

#State

The transaction pool worker uses a clone of the state from the main thread. All changes are synced from the main thread to the transaction pool worker as the delta changes when a new block is committed. This keeps the states in sync, ensuring transactions are verified against the real state data.

To improve boot time, state snapshots are used if found locally. Both the main and worker threads restore the state from the same snapshot.

#Worker Communication

The main process controls the transaction pool worker process via IPC methods, also known as handlers. Various handlers are implemented:

• Start: Reads the transaction pool database and attempts to re-add known transactions.
• ImportSnapshot: Imports a snapshot of the state relative to the height passed in the parameters.
• Commit: Triggers on block commit and syncs the new block and state deltas to the worker. Milestone changes are handled accordingly.
• GetTransactions: Returns the list of transactions that are ready for forging. Transactions are listed by priority and limited by the blocks maxTransactions value.
• SetPeer: Sets a new transaction pool peer. The peer was previously verified on the main thread. The worker keeps a local peer list to broadcast transactions.
• ForgetPeer: Removes unresponsive or malicious peers from the peer list.
• ReloadWebhooks: Reloads webhooks from the database. This handler is triggered when there is a change in the main thread subscriptions.

#Transaction Pool API

The transaction pool API also runs inside the worker. This setup ensures that the main process is unaffected by high activity and long response times in the transaction pool.

Unconfirmed transactions are no longer synced to the PostgreSQL database and cannot be queried through the public API. Instead, clients must post transactions directly to the transaction pool API as before and can now query them on the same API. Here is a list of exposed endpoints:

• POST /transactions: Submit a new transaction.
• GET /transactions/unconfirmed: Retrieve all unconfirmed transactions.
• GET /transactions/unconfirmed/{id}: Retrieve a specific unconfirmed transaction by ID.
• GET /transactions/types: Get a list of transaction types.
• GET /transactions/schemas: Get transaction schemas.

#Proposal Processing

We’ve implemented several optimizations to enhance the speed of consensus and block processing.

Now, when proposals arrive, they are only partially deserialized. We limit deserialization to the essential data required for basic proposal verification, omitting the block deserialization. This targeted approach significantly accelerates the verification process, particularly for blocks carrying large payloads.

During our testing, we observed that block deserialization could take up to 2 seconds. We have now moved this process to a later stage, significantly reducing the time required to approximately 200 milliseconds. As a result, nodes can broadcast proposals more swiftly, enhancing overall throughput.

This rapid dissemination of new blocks is crucial for consensus, as most nodes must receive and relay new blocks promptly, leveraging the epidemic broadcast mechanism at the P2P level. Since a Mainsail node does not broadcast to all other nodes in the network, but only a subset of them, we had to reduce the re-broadcast time between nodes to accommodate for this.

Blocks with large payloads can take several seconds to fully deserialize. Previously, the main thread was blocked during this time. To fix this, intentional timeouts were added during deserialization to release the main thread and handle other requests, like P2P requests, to keep the node connected and responsive.

To improve block creation, the forging node skips the deserialization step. Unlike the previous implementation, the forging node no longer needs to serialize and deserialize the block: as the creator of the block, it can trust its data.

For larger blocks, this can result in an improvement of about 2 seconds, allowing new blocks to be provided to the network earlier and offering a wider window for block preparation.

#Consensus Stages

To customize the network, some properties can be adjusted in the milestones. The timeouts section includes the following options:

• tolerance: Time tolerance between nodes in case of time drift. The default is set to 100ms. This is currently used for checking future blocks but can be extended to other areas as well.
• blockTime: Minimum delay between two consecutive blocks. The default is set to 8s.
• blockPrepareTime: Time assigned to allow for forging a block. Users will not notice the difference when consensus stages are executed quickly and completed well before the blockTime. This value ensures that the forger has enough time to process a block even when consensus is not completed within the blockTime. Especially under high load, this results in a more stable block size.
• stageTimeout: Time allocated for each consensus stage before triggering the next stage. Examples:
  • proposalTimeout triggered after the round starts.
  • prevoteTimeout triggered after +2/3 of prevotes are received.
  • precommitTimeout triggered after +2/3 of precommits are received.
• stageTimeoutIncrease: Increases stageTimeout by this value each round without block commit. On block commit, stageTimeout is reset to default.

Block processing was slightly changed compared to the initial consensus implementation. The blockPrepareTime was added, allowing selected proposers to prepare a block right after the commit, even if the new round has not started yet. This uses previously idle time for block creation and transaction validation on the forging node. This change also ensures enough time in case of long consensus cycles (when consensus takes more time than the blockTime value) and can handle larger blocks better.

Check the following diagram to get a better understanding of how consensus operates under different scenarios:

#Summary

The latest changes bring improvements to Mainsail, resulting in higher throughput, more resilient nodes, and a stable consensus process. In the latest performance test on a network with 51 nodes, we were able to process ~ 442 multipayments with 256 recipients. This is higher compared to ARK’s current production network, which handles 150 multipayment transactions per block with a maximum of 128 recipients.

We will continue working on performance improvements in the future along with EVM support.