It has been a bit over a month since our last technical progress update on Core v2. We have made tremendous progress preparing for MainNet and after recent events have refined our new timeline for release.
Let’s get started!
First, we need to cover both good and bad news. The bad news is, this will be the last technical progress update before MainNet v2 launch. The good news is, the next post will be the announcement (yes with an actual date) of the official launch of ARK V2 on MainNet!
We have arrived at a point where all of the features and optimizations have been integrated into v2. The new Core about to be released is as we envisioned it — fast, efficient, stable, modular and ready for the world to build upon.
As we are ahead of schedule, we will be conducting our final phase of testing on DevNet, starting Monday, October 29th. This will cover:
- Stress Testing (trying to break the network by mass spam transactions, finding limits, massive voting and unvoting aka delegate shifts, making sure AIP11 transactions are not yet accepted, …).
- Snapshots / Forking and miscellaneous (learning to rebuild from local snapshots, forking and recovering after, and other tests requested by the community).
If you want to join the testing on DevNet, don’t hesitate to join Slack (channel #devnet) and our team will be there to address any questions and guide you through setting up the testing environment.
Changes In Numbers
Since the last technical blog post there have been:
- 17 different developers contributing to the Core.
- 175 pull-requests to the Core repository.
- 688 files changed in the Core repository.
- 25,302 lines of code added to the Core.
- **26,061****lines of code deleted** from the Core.
Features, Refactoring and Other Commits
New snapshot system — one of the drawbacks in v1 is the reliance on snapshots provided by ARK. As such, we have re-designed the entire snapshot process in the new Core to further decentralize ARK and provide a better and more resilient system. The main purpose of the new snapshot plugin is to provide local snapshot functionality for the node operator. This allows the node operator to rebuild locally from their own data in case of issues with the blockchain database. The plugin also provides a CLI interface with the following available commands:
- create (with options to append or to select start and end block heights to export)
- import a snapshot
- verify
- and rollback (which will also backup forged transactions during the process).
The snapshot system will be further developed as we move forward. It will provide an option for faster downloading of blocks which will allow trusted nodes to establish a continual stream of data between themselves. By doing so, a delegate or node operator can establish his own snapshot and rebuild from locally verified data. In case of emergencies, local rollbacks will be made easier as the chain can be reverted to a previous state, while backing up the forged transaction to be re-sent to the transaction pool later if necessary.
New faster in-memory transaction pool— one of the biggest changes made over the previous months was to replace the Redis transactions pool with a hybrid one that stores all data and serves all queries from memory. From this location it occasionally syncs/saves to a local SQLite database. This not only increases performance, but also provides improved stability and handles transactions better as they hit the pool. It is also more future proof if network load increases.
Transaction pool sort by fee and limitations —before this implementation there was no sorting of transactions by dynamic fees. The sorting order needs to be taken into account when retrieving transactions from the pool. Now, the sorting is prioritized by fee, with higher fee transactions entering the block sooner.
BigNumber integration — BigNumber.js was implemented because JS has a limit to how large a number can be. When you exceed that limit, rounding and modifications occur to make the number fit inside that limit. Also, if you have “123”+“123” in JS you will get a result of “123123”instead of “246”, since the numbers are treated as strings and connected together rather than added. Now, by using BigNumber.js, we can avoid unpredictable behaviors with like numbers being treated as strings, having decimals, being floating numbers and exceeding the number limit JS has. With this new implementation we are not only future-proof, but we can now trust that all mathematical operations yield correct results even for very large numbers.
New improved HDWallet implementation — ArkJS uses the HDNode class to deal with HD wallets. It originated from the bitcoinjs-lib, but eventually was removed upstream. The new way to deal with HD wallets is to use the BIP32 module. A few features are not provided by BIP32, including the ability to convert a keypair to a node and vice versa. This new implementation adds a utility class HDWallet with a few convenience functions to get a BIP32 node or turn one into a keypair. Which in effect also provides new functionalities as outlined in AIP 20 .
Build delegate list from memory — before this implementation the calculation of the delegate list was made from the database. This degrades performance because the wallets that changed during the last round should be synced with the database before the calculation occurs (using a very costly upsert). This has now been replaced with an in-memory implementation which is not only faster, but more reliable and sorts some of the edge cases that could occur on v1.
Message sign and verify — added new implementation of message signing and verification in the Core as needed by the new desktop wallet.
Remote API authentication —enables the P2P remote API and fixes the authentication via whitelisting to avoid accepting requests to the internal and remote APIs as a new peer.
Optimize vote balance calculation —we changed how vote balances are calculated. Instead of calculating them from scratch all the time, a “delta update” approach is used. Additionally we changed the order of buildVotes and buildDelegates during SPV. By doing so the delegate vote balances can be easily validated.
Use secp256k1 for verifying and signing — before this implementation we had ECPair, ECPoint and ECSignature, re-implementing a lot of ECC in JavaScript. But with integrating secp256k1 we can do all of that as well, natively, removing the need to have our own slow ECC implementation. By using native secp256k1 the time spent verifying and signing is cut down substantially. It is approximately more than twice as fast as before, average time of verify (before: 1.56ms | after: 0.67ms).
Refactor crypto package —we have refactored and cleaned up the @arkecosystem/crypto package. Most of the classes carried over from bitcoinjs (ECPair, ECPoint, ECSignature, …) are not needed and only add complexity without value and can be removed.
Changes include:
- Removed ECC classes
- Removed other obsolete classes and files
- Lesser dependencies
- New functions keysFromWIF, keysToWIF, signHash and verifyHash
An interesting side effect is that all unit tests now complete much quicker (in under 2 minutes on local machines from our devs, as compared to previous 5–6 minutes).
Database rollback — introduces a simple rollback mechanism to restore database integrity. It looks at the top height and rolls back X blocks Y times and tries to verify the blockchain each time until it either succeeds or fails for good. It works reliably if the damage is recent, and databases beyond repair will make a use of snapshots.
Database verify check — added additional database checks for the integrity of the DB, such as checks for negative balances.
Performance increase on batch wallets on insert — by properly batching the wallets when doing a force insert we cut down the time spent considerably (average example : before - saveWallets: 2063.485 ms | after - 1266.543 ms).
Made database calls during applyRound non-blocking — with the new implementation of in-memory calculations, we got rid of the remaining blocking db calls in applyRound. (total before: applyRound: 271.226ms | total after: applyRound: 80.579ms).
Optimize Transaction.fromBytes — we have reduced calls on one of the operations which in effect speeds it up by 2x. As a test of parsing 150,000 transactions on devnet (Before: 52601.147ms | After: 25641.869ms).
Guard add limit one per vote in pool — adding limit functionality to pool guard. Limit per sender to have only 1 vote/unvote transaction allowed in the pool. This is related to dynamic fees sorting and less mistakes on the client users.
Refactored plugin extensions — we added an extends property for plugins which can be used to extend interfaces like the database and transaction pool packages as it was kind of ineffective to register them via configuration without them accepting a configuration.
Sign transaction with WIF —this adds the ability to sign transactions using a WIF. This is needed, because BIP38 decodes to a WIF.
Search by owner ID — added support for the ownerId property which allows to search for transactions by sender and recipient.
New plugin Core Debugger CLI —testing was quite difficult when something with blocks or transactions failed and you wanted to see what is behind the hex string. This plugin adds some basic CLI to perform serialize, deserialize, verify and more to come.
New package Core Utils — moved some commonly shared utilities into a package core-utils which can be re-used by other packages and reduces duplication at the same time.
CircleCI— we have replaced Travis with CircleCI config to execute core tests with node 9 and 10, along with lint and depcheck (node 10). From the tests we made with CircleCI it is way faster than Travis and allows parallelization so that we can have the 4 jobs (tests node 9 / node 10 / lint / depcheck) running at the same time. Tests showed 5 time reduction time from Travis running for 15 minutes to new CircleCI running only 3 minutes for the same PR.
Block exceptions — added the ability to add exceptions for blocks simply via config file.
Enable block height caching —fixed integration of caching of the block height so confirmations can be viewed properly.
Replace LevelDB with KeyV— LevelDB was very error prone on Travis and would randomly get LOCK issues. Another positive side-effect of KeyV is that it supports custom drivers for things like Redis, SQlite and Postgres.
Create table util —small helper to create readable tables for testing via .
Transaction guard error feedback — the transaction guard now stores the errors which marked a transaction as invalid. Those are then used in the response of the respective handlers.
Handle wrong nethash suspension — now properly handles requests on the wrong network with a suspension.
Remove Sequelize — we have removed core-database-sequelize which can now be found here as we focused on native PostgreSQL integration to optimize database performance.
Forged rewards and fees for delegate (v2 API) — new API v2 feature which adds the forged rewards and fees to the delegate transformer.
Only trigger forging events after a successful new block broadcast — by changing the order of the code, the events ‘block.forged’ & ‘transaction.forged’ should only be fired if the forger successfully broadcasts the new block.
Refactored transaction serialized to hexstring — this implementation aligns the type of tx.serialized with block.serialized and changes it from a Buffer to a hexstring. Most places called toString('hex') on tx.serialized and this change allows to drop some code which was needed, because Block and Transaction were inconsistent.
Replace toString with toFixed — replaced all toString and toNumber calls with toFixed so that we won’t run into issues like on v1.
Temporary 503 ban — we added a very short 503 ban without any weight penalty that could result in a longer ban. 503 happens when we get “blockchain not ready” message, which means the node is syncing or rolling back.
Log which transactions failed to verify — when a block contains invalid transactions, also log the serialized transaction.
Add retry delay when calling __chooseHost on forger startup — When starting a forger and relay simultaneously, the forger tries to connect to the relay immediately, even though it is not ready. We added a small delay, so that calling __chooseHost during forger startup doesn’t flood the poor relay with requests.
Find peers timeout — Fixed an issue when getting peer list response & also adds a timeout of 5 seconds to each request, in case the peer is not responsive.
Fixed wrong order of indexing round — this solved an issue on slow rebuild for our test servers.
Handle common block bans — better handling of the “Unknown” ban scenario.
P2P header validation — validates headers and removed unused validation.
Testing Suite Updates
- Added core-blockchain tests.
- Fixed core-database revertRound which was not working for reverting round 2.
- Fixed core-blockchain isSynced which used an incorrect height property.
- More core-p2p tests.
- Fixed call queueBlock with simple block object, not Block instance.
- Use a mock server to test forger Client http calls.
- Increased coverage of core-forger/manager tests.
- Transaction builder verification.
- Updated jest config to test all packages + Travis config fix for core-json package.
- Refactored core-test-utils transactions generators.
- Double spend tests on api, p2p, transaction-pool-redis. (+ minor refactor on core-test-utils)
- Integrating code coverage in CircleCI build.
- Added a missing test for the format timestamp helper.
- Tests for emptying wallet tests for pool-mem, core-api and core-p2p.
- Replaced toBeTruthy / Falsy by toBeTrue / False when appropriate.
- Update core-api test setup + core-p2p ntp mocking for CircleCi.
- Reduce nondeterminism in the tx expiration test.
- Extend test coverage on mempool.
And a lot of other small bug fixes and other improvements throughout the Core — a never ending process!
We are beyond happy with our achievements thus far and can now say this is the release candidate for MainNet! You can expect another blog post soon with the announcement date for moving v2 to MainNet.
If you would like to get into development of Core and earn some ARK please give our GitHub Development Bounty blog post a read.
- ARK Core repository :
- ARK Core docs :
- ARK Core open issues:
