ARK Core: Bi-Weekly Progress Update

It’s been two weeks since our last core update, which means it’s time for another update! In this update we will explain what we’ve been working on and our technical progress with ARK Core v2 over the last two weeks.

One of the biggest changes we’ve made is the integration of a native and pure PostgreSQL database layer that will not only improve overall performance of Core (on some instances tenfolds), but leaves us with a lot of room for additional improvements and lets us interact with the database in a more native way. Similar to the way we have implemented PostgreSQL, we’ll make additional native plugins for other databases like SQLite, which over time, will eliminate the need for Sequelize.

The second biggest area of focus has been on eliminating memory leaks when rebuilding a node. The current process didn’t commit blocks to the database, but had them stored in memory, which was getting flooded with tens of thousands of blocks just piling up.

We are also continuing with testing, optimizing and fixing bugs as they are reported. So far DevNet has been running stable for a month and we’ll slowly start preparations for MainNet upgrade soon.

The rest of this post gets more technical by going over code fixes, changes and additions.

#Changes In Numbers

Over the course of last 14 days there were:

• 9 different developers contributing to the Core.
• 32 pull-requests to the Core repository.
• 279 files changed in the Core repository.
• 4,076 lines of code deleted from the Core.
• 8,191 lines of code added to the Core.

#Features, Refactoring and Other Commits

• Pure PostgreSQL integration — We integrated PG-Promise which provides superior performance and resolves some of the problems we were facing with Sequelize. This paves the way for future optimizations on the database layer without having to do any compromises because we have to keep in mind how some other databases work. The performance impact on the API has already seen tremendous improvement after some initial testing as count queries are now possible even on low-end servers which allows us to provide end-users with better pagination of data and serve that data faster.
• Modernise the P2P API — this has been one of the biggest changes in the last two weeks. This means we now make proper use of status codes, have standardised responses and middlewares that take care of making sure things like the blockchain and transaction pool are mounted before any requests hit the API. Some of the changes are: restructured internal P2P API, remote P2P API and public P2P API. Removed try catch noise, added joi schemas, new transaction pool ready plugin developed, integrated validation schemas, adjusted forger to new P2P API, renamed public P2P to peer P2P, added badState helper to __analyseNetworkState, moved config API to its own version, removed leftovers & bad references. Handle request acceptance error, replaced error responses with boom, adjusted queue logging, adjusted P2P server tests, use monitor directly in v1 API and fixed wrong status code & response format.
• Transaction pool dynamic fees —as it was hard to identify what is happening with dynamic-fee transactions when not accepted by local node, but forwarded to other nodes, the logic has changed a bit too:

1. If transactions are not accepted by the delegate’s fee criteria, it still goes into the pool, where it can be watched via
2. If it will be forged by other delegates, you will no longer find it there, otherwise it will wait in the pool for 6 hours and then it will expire.

The transaction-pool-interface was also cleaned up and all Redis things moved down to the appropriate place.

• Drop ArkJS dependency — since this library is officially deprecated and most of it’s functionality was already in the @arkecosystem/crypto package we added the missing parts (namely getBytes/verifying/signing) and removed the dependency completely. All other core packages have been updated to use @arkecosystem/crypto instead of arkjs. This is necessary to apply future improvements to the crypto package.
• Added an option for client to override peers — this gives the client package the ability to work with other networks. Passing in the network name previously meant that only mainnet and devnet were possible.
• Improve wallet manager method names — some methods were renamed, as having the prefix getWallet is redundant if the methods are inside the WalletManager. Also included a small refactor for more consistent behavior of trigger the wallet:cold:created event as a method was rendered ineffective after some renaming and looking at the logic.
• Added Docker compose for MainNet — easier for developers to start and switch configuration when working (devnet, testnet, mainnet).
• Test suite for JSON-RPC package — fixed signing of transactions after filling in the recipientId and amount, make database operations async and add close() method. Fixed tests and added core-json-rpc package to jest config and added core-p2p and axios-mock-adapter dev dependencies.
• Test suite for Core-API package — fixed core-api broken tests and used Hapi request injection instead of axios, fixed a bug in core-database-sequelize where queries with condition ‘column = 0’ were wrongly interpreted, fixed a bug in core-api wallet handler, deleting request.params.id so that it does not produce a wrong SQL query and added a new method to stop forger manager + deregister plugin method, useful in tests afterAll.
• Use Hapi inject instead of axios for server requests —for tests in Core P2P package, use Hapi inject instead of axios for server requests.
• Refactor GraphQL by migrating to Apollo server 2 — the GraphQL service was not starting properly, this was refactored with the implementation of apollo-server and fixed the requests made to the db.
• Initial tests on GraphQL package — initial tests on GraphQL package were added and fixed wallets query as it was not filtering by vote.
• Adding crypto package — adding crypto package and testing if data.timestamp exists.
• Reuse of transactions — reuse the created transaction instead of creating a new one.

#Fixed Bugs

• Process queue memory leak — removing async where not needed, fixed memory leak when adding pause on processQueue to empty it when syncing and enabled process queue to finish processing before adding new payload.
• Fixed validation of incoming payloads —in ‘WalletManager’ there was a bug allowing possible 0 fee transactions being accepted and sent to cold wallets. Another check allowed to send votes for non existing/not registered wallets and in case of cold wallet, still check transaction conditions for vote.
• Listen for all events on Webhooks — Webhooks were only listening for specific events, now they will listen for all available events.
• Fixed suspending peer object — Suspend(peer) method was crashing as the peer sent to the method was not a Peer object, but just peerreceived from accept-request. Moved the object creation higher and passing Peer object to the suspend method.
• Added missing fields to transaction object — transactions returned from the /api/transactions endpoint were missing signSignature and signatures fields.
• Save only dirty wallets — save only modified wallets during the last round set to false, saving only modified wallets.
• Properly set crypto config on launch — crypto config was not setup properly when starting the Core process.

With each step we are closer to putting Core on MainNet and the whole ARK team is happy with the way things are developing and the current progress being made.

#How Can I Help With Development?

If you want to help, get involved and earn some extra ARK be sure to give ARK GitHub Development Bounty a read and get coding:

• ARK Core repository :
• ARK Core open issues :https://github.com/ArkEcosystem/core/issues