In a previous post, I proposed an approach to improve the transaction list in the wallet.
TLDR: we are making progress, in this first post I propose a bunch of additional optimizations based on experience gained from implementation, and an algorithm to solve the fetching transaction history problem.
We started the work last week and here are the parts that are already done:
- implementing a subscription mechanism through signals to avoid the pull loops of web3 and in the wallet (done by @igor)
- use subscriptions for updating transactions confirmations dynamically (done in https://github.com/status-im/status-react/pull/8158)
- use subscriptions for adding new token transactions dynamically (done in https://github.com/status-im/status-react/pull/8184)
The next step is to fetch transactions once at login from etherscan and then check in each new block if there is some eth transaction to get rid of the last wallet loop.
Transaction history for regular ethereum transactions is not a solved problem and the only solutions I could find were either to use a third party service like etherscan or iterate over each block looking for transactions.
eth_newBlockFilterreturns only the hash of the last block, which requires an extra call to get the actual block
@igor : possible optimization would be to make a subscription that returns the last block number instead of hash. Since this is also the only way to get last ethereum transactions we could also make a subscription that only signals transactions to and from and address.
Token transaction history can be fetched through eth logs but the rpc call can timeout if the range is too high so it needs to be done iteratively.
We have two problems to solve:
- fetching the complete token transaction history (currently limited to 2 weeks, not available from etherescan api, though they have an ethlog equivalent that could be checked out)
- fetching ethereum transaction history without etherscan
I propose the following algorithm composed of 2 steps to solve both issues
Find all ethereum transactions without etherescan
This part is optional, we could use it only when etherescan is down, for instance during chaos unicorn days.
The JSON RPC API contains the following calls:
- eth_getBalance which returns the eth balance of an address at a certain block
- eth_getTransactionCount which returns the number of transaction from an address up to a certain block
Unless I am mistaken, we should be able to find all inbound and outbound eth transactions from block 0 with a recursive dichotomic search that would be at most log2(last-block) (roughly 23) levels deep.
- check the number of transactions and balance at current block (block x)
- check the number of transactions and balance at last checked block (block y) (genesis for the first run)
Algorithm: input balance diff and transaction count diff:
- if balance and number of transactions have not changed: there was no transactions, we stop
- if balance has increased but number of transactions have not changed: we are looking for inbound transactions (count transactions is only for outbound transactions), we check balance at block (x+y)/2 and repeat the algorithm for both [x (x+y)/2] and [(x+y)/2 y] ranges with transaction count diff 0.
- if both have changed: we are looking for outbound transactions (even 0 eth transaction cost gas), we check balance at block (x+y)/2 and repeat the algorithm for both [x (x+y)/2] and [(x+y)/2 y] ranges with transaction count diff 0
Everytime we hit a range of 1 block we use
eth_getBlockByNumber to get the block and retrieve the transactions from and to the account address.
The implementation of this part of the algorithm could be bountied, and eventually implemented in status-go entirely for efficiency.
Find all token transactions
Infura has some difficulties with huge ethlog ranges which is why this algorithm tries to minimize the amount of calls required to find all token transactions.
Some of the transactions found by the first step are outbound token transactions. It can be determined by fetching their transaction receipts.
At this point all we are missing are the token inbound transactions. For those we are first going to check the balance of the shown tokens.
From there we have 2 options:
- We can then use a similar approach as for regular transaction with the
balanceOfmethod of the token contract, which takes an address parameter. eth_call takes an additional block parameter.
- For each token transfered we do a dichotomic search using balance of to find the first inbound transfer between block 0 and block(first transfer)
- For each outgoing token transfer, we check the balanceOf at the block the transfer happens for the token transfered. If balanceOf at block(transfer n) is different from balanceOf at block(transfer n-1) + amount transfered, we do a dichotomic search between the 2 blocks (or call ethlog if it is estimated faster, less than 100 000 blocks)
- We will then use that balance and the known outbound transactions to find the ranges in which the inbound transactions have occurred.
- We will then scan these ranges using ethlogs. After this step only the inbound transactions that occurred before the first outbound transactions for each token will be missing.
- For those we will call ethlogs iteratively going backward from the block of the first known outbound transaction back to the genesis block, except we stop once the balance has reached 0.