How to configure electrs to retrieve large address histories

[cricktor]

Hi,
I'm running Bitcoin Core v0.21.1 and electrs with Raspiblitz v1.7.0 as my personal node and Electrum server on a Raspi 4B with 8GB RAM.

Sometimes I like to do some blockchain digging and analysis and run into problems when a big address history needs to be reported back to the Electrum client or local Bitcoinexplorer/mempool.space.

I tried to find some configuration clues, but wasn't very lucky. The local installed tools as like bitcoinexplorer.org and mempool.space request the address history from the electrs server. (It's configured to do it.)

I don't mind if the Raspi has to do some deep data digging, it should've enough RAM. Can I configure electrs such that it would retrieve big address histories like some thousands of transactions?

Any advise is very appreciated! If I missed to provide some helpful further details, let me know I will amend it.
bitcoind has txindex and blockfilterindex active and is of course fully synched.

[cricktor]

In ~electrs/.electrs/config.toml I bumped txid_limit to 100000 or more but I still have timeout issues with official Electrum v4.1.5 wallet when it tries to synchronize a watch only wallet of some bitcoin burn addresses that I have collected (don't have some better address examples right now; those burn addresses don't have such a huge transaction history, so I don't really understand why it causes such sync issues).
My local Electrum wallet synchronized only with my own electrs server.

[Kixunil]

Hi, the current Electrum protocol is not great when it comes to queries with many results, so we will have to wait for the new proposal to add pagination. In the meantime consider trying p2p branch which contains some optimizations and may work better. It'll become a new version somewhat soon anyway.

I tried to find some configuration clues, but wasn't very lucky.

Did you read the documentation? It explicitly points out there's --help page.

[romanz]

You can increase the default limit using the --index-lookup-limit flag (currently supported on p2p branch):

electrs --network bitcoin --db-dir ./db --daemon-dir ~/.bitcoin --index-lookup-limit 1000

[romanz]

Raspiblitz 1.7.1 runs electrs 0.9.0, which should have better performance :)

[MCM-Mike]

Is there an update on:

You can increase the [default limit](https://github.com/romanz/electrs/blob/b916c2802bdee5b9c98b420d2a93dada3cdf055e/internal/config_specification.toml#L96) using the --index-lookup-limit flag (currently supported on p2p branch):

electrs --network bitcoin --db-dir ./db --daemon-dir ~/.bitcoin --index-lookup-limit 1000``` 

and pagination ?

[cricktor]

@Kixunil Take e.g. Satoshi's receiving address of the Genesis block: it received a lot of donation and dust transactions.
Response from mempool.space explorer is:

Address
1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa

Error loading address data.
The number of transactions on this address exceeds the Electrum server limit

Consider viewing this address on the official Mempool website instead:
https://mempool.space/address/1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
http://mempoolhqx4isw62xs7abwphsq7ldayuidyx2v2oethdhhj6mlo2r6ad.onion/address/1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa

(>200 index entries, query may take too long)

RPC-BTC-Explorer mumbles:

Error retrieving transaction history (#1)
{
    "e": {
        "code": 1,
        "message": ">200 index entries, query may take too long",
        "userData": {
            "address": "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa",
            "sort": "desc",
            "limit": 10,
            "offset": 0
        }
    },
    "status": "rejected"
}

[MCM-Mike]

... Multiple addresses should be doable with multiple requests but it may take some time.

Thanks, I do know about best practice behavior for BTC addresses.
But leading with a "good" example are exchanges who are reusing same "deposit" addresses for their clients, etc and sometimes even "withdraw" addresses. This is why it's sometimes important to have all transaction's of a single address (native)-segwit, legacy and now taproot).

(could lead a bit OT)
@cricktor can you use "electrs" with btc-rpc-explorer, what are your settings there in terms of enviroment variables for BTCEXP_ADDRESS_API ?

[Kixunil]

I see, the error message is reasonable and yes, it's the lack of a suitable configuration of electrs.

Frankly, I don't think it's much of a privacy issue to lookup these public addresses in public explorers. Even if an explorer was selling the data of customers to chain analysis (like already happened) if they are stupid enough to think it's your address it actually helps the ecosystem.

Anyway, exchanges reusing addresses is nuisance that's true. However maybe not optimizing it puts pressure on them to stop doing stupid shit?

[cricktor]

... can you use "electrs" with btc-rpc-explorer, what are your settings there in terms of enviroment variables for BTCEXP_ADDRESS_API ?

Likely relevant settings in RaspiBlitz's node configuration env-file for btc-rpc-explorer:

BTCEXP_BITCOIND_RPC_TIMEOUT=10000
BTCEXP_ADDRESS_API=electrumx
BTCEXP_ELECTRUMX_SERVERS=tcp://127.0.0.1:50001

Frankly, I don't think it's much of a privacy issue to lookup these public addresses in public explorers. Even if an explorer was selling the data of customers to chain analysis (like already happened) if they are stupid enough to think it's your address it actually helps the ecosystem.

If I'm dealing with addresses that are not related to me in any way, I wouldn't mind using suitable public explorers. But what kind if argument is yours? If I want to keep privacy for blockchain analysis which does touch my own addresses, I need my local node and tools setup to be able to deliver the data properly. And this includes a suitable configuration for electrs that delivers in the best case any address history, even if it takes some time and delays. If electrs is not the tool for this, I'm fine with that: then I need to find a more suitable tool. If I need bigger indices for electrs to solve it, I'm fine with that too, then it's my decission to make it work.

Currently I'm just not yet aware of where and what to tweak.

[Kixunil]

Yes, use your own node for your own stuff. If you don't reuse addresses (which would undermine your goal anyway) electrs should be fine.

You may want to look into blockstream for of electrs if you want to reuse addresses. It's more performant but eats much more space.