Bug: Unable to sync, failed to get block

[asyscom]

Hello,
I can’t sync electrs. Every time I launch the program, after a variable time between 10 minutes and an hour, the sync gets stuck and I receive this error:

Error: electrs failed

Caused by:
    0: sync failed
    1: failed to get block 00000000000000013c7b5580791075c98fa0bd2506ed777d9362033b49790cf5
    2: receiving on an empty and disconnected channel

If I restart the Bitcoin service and then electrs resumes syncing, but after a while, it gets stuck again. Obviously, I can't keep restarting Bitcoin continuously. The node works correctly, including integration with LND, ThunderHub, and LIT.

Electrs version
Electrs version 0.10.5

System running electrs
SO: Ubunu Server 22
Bitcoin: version v27.1.0
LND: 0.18.2

Thanks in advance
Davide

[romanz]

Please make sure your bitcoind is configured with the following flags:
https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
[email protected]

[asyscom]

Please make sure your bitcoind is configured with the following flags: https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
[email protected]

Thanks, i've update my bitcoin,conf
#datadir=/var/lib/bitcoind

bitcoind configuration

some values might be overruled directly systemd-service exec call parameters

mainnet/testnet

testnet=0

Bitcoind options

server=1
daemon=1
txindex=1
disablewallet=0
peerbloomfilters=1
datadir=/var/lib/bitcoind
main.debuglogfile=/var/lib/bitcoind/debug.log

Connection settings

rpcuser=xxxxxx
rpcpassword=xxxxxxx
main.rpcport=8332
test.rpcport=18332
rpcallowip=127.0.0.1
main.rpcbind=127.0.0.1:8332
test.rpcbind=127.0.0.1:18332
main.zmqpubrawblock=tcp://127.0.0.1:28332
main.zmqpubrawtx=tcp://127.0.0.1:28333

Raspberry Contabo

dbcache=2048
maxorphantx=10
maxmempool=500
#maxconnections=400
maxuploadtarget=5000
onlynet=ipv4
onlynet=onion
#Tor
proxy=127.0.0.1:9050
listen=1
listenonion=1
bind=127.0.0.1
externalip=my-onio
#custom
externalip=my-ip
maxconnections=N
[email protected]

After I restarted bitcoin, lne and electr the sync resumed for a 10 minutes and then gave the same error again

Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 }
[2024-08-16T09:44:04.582Z DEBUG tiny_http] Server listening on 127.0.0.1:4224
[2024-08-16T09:44:04.582Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
[2024-08-16T09:44:04.582Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
[2024-08-16T09:44:04.582Z DEBUG tiny_http] Running accept thread
[2024-08-16T09:44:04.611Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 76 SST files, 0.63497689 GB, 0.048874206 Grows
[2024-08-16T09:44:04.611Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 })
[2024-08-16T09:44:04.908Z INFO electrs::chain] loading 214350 headers, tip=000000000000019b729b686a16de4c46da980c9fd8709043de36b39c1dee820a
[2024-08-16T09:44:05.165Z INFO electrs::chain] chain updated: tip=000000000000019b729b686a16de4c46da980c9fd8709043de36b39c1dee820a, height=214350
[2024-08-16T09:44:05.168Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-16T09:44:05.169Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T09:44:05.169Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T09:44:05.170Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-16T09:44:05.171Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T09:44:05.175Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723801445, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 2267302671146926227, user_agent: "/Satoshi:27.1.0/", start_height: 857018, relay: true }
[2024-08-16T09:44:05.178Z DEBUG electrs::p2p] got 2000 new headers
[2024-08-16T09:44:05.179Z INFO electrs::index] indexing 2000 blocks: [214351..216350]
[2024-08-16T09:44:05.179Z DEBUG electrs::p2p] loading 10 blocks
[2024-08-16T09:44:05.230Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed
[2024-08-16T09:44:05.232Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected
[2024-08-16T09:44:05.232Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin
[2024-08-16T09:44:05.232Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send
Error: electrs failed

Caused by:
0: sync failed
1: failed to get block 0000000000000247f6cc4f073a9b38a770566d51e03bdb588cf1b7fa5ac41782
2: receiving on an empty and disconnected channel

[asyscom]

Please make sure your bitcoind is configured with the following flags: https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
[email protected]

ive triedto remove maxuploadtarget and restart all services but after some minute riceived the same error
Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 }
[2024-08-16T11:03:40.580Z DEBUG tiny_http] Server listening on 127.0.0.1:4224
[2024-08-16T11:03:40.580Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
[2024-08-16T11:03:40.580Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
[2024-08-16T11:03:40.581Z DEBUG tiny_http] Running accept thread
[2024-08-16T11:03:40.607Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 326 SST files, 1.216418774 GB, 0.096343168 Grows
[2024-08-16T11:03:40.607Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 })
[2024-08-16T11:03:41.002Z INFO electrs::chain] loading 245120 headers, tip=00000000000000672b757cf1acd624a0a9d5bef7b641929b5e270ad8fd9b4af8
[2024-08-16T11:03:41.317Z INFO electrs::chain] chain updated: tip=00000000000000672b757cf1acd624a0a9d5bef7b641929b5e270ad8fd9b4af8, height=245120
[2024-08-16T11:03:41.325Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-16T11:03:41.326Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T11:03:41.326Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T11:03:41.327Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-16T11:03:41.327Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-16T11:03:41.329Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723806221, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 14602067919111189498, user_agent: "/Satoshi:27.1.0/", start_height: 857029, relay: true }
[2024-08-16T11:03:41.334Z DEBUG electrs::p2p] got 2000 new headers
[2024-08-16T11:03:41.335Z INFO electrs::index] indexing 2000 blocks: [245121..247120]
[2024-08-16T11:03:41.335Z DEBUG electrs::p2p] loading 10 blocks
[2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed
[2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected
[2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send
[2024-08-16T11:03:41.386Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin
Error: electrs failed

Caused by:
0: sync failed
1: failed to get block 0000000000000050b60fecadb2146fd594db80b5bca34ec03862d3caa2a53523
2: receiving on an empty and disconnected channel

[Kixunil]

I assume maxconnections shouldn't be lieral N but some high number (100?).

[asyscom]

I assume maxconnections shouldn't be lieral N but some high number (100?).

Hum I meant that I can put N because it's for infinite.
Need i put a number?

[Kixunil]

Check bitcoind's doc.

[asyscom]

Check bitcoind's doc.

OK, tomorrow I'll set it to 200.
Let me time to try and give a feedback

[asyscom]

Check bitcoind's doc.

nothin to do, i'se to 400 maxconnections, removemaxuploadtarget, set [email protected] but the errors is still the same
Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 }
[2024-08-18T10:06:47.126Z DEBUG tiny_http] Server listening on 127.0.0.1:4224
[2024-08-18T10:06:47.127Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
[2024-08-18T10:06:47.127Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
[2024-08-18T10:06:47.127Z DEBUG tiny_http] Running accept thread
[2024-08-18T10:06:47.156Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 363 SST files, 1.818754212 GB, 0.144986034 Grows
[2024-08-18T10:06:47.157Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 })
[2024-08-18T10:06:47.642Z INFO electrs::chain] loading 275880 headers, tip=0000000000000003938261a92c469e8474754d6a23b1e5ed1219159c50b0a200
[2024-08-18T10:06:47.999Z INFO electrs::chain] chain updated: tip=0000000000000003938261a92c469e8474754d6a23b1e5ed1219159c50b0a200, height=275880
[2024-08-18T10:06:48.007Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-18T10:06:48.007Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-18T10:06:48.008Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-18T10:06:48.009Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-08-18T10:06:48.009Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-08-18T10:06:48.012Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723975608, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 4227069755078880485, user_agent: "/Satoshi:27.1.0/", start_height: 857324, relay: true }
[2024-08-18T10:06:48.015Z DEBUG electrs::p2p] got 2000 new headers
[2024-08-18T10:06:48.016Z INFO electrs::index] indexing 2000 blocks: [275881..277880]
[2024-08-18T10:06:48.016Z DEBUG electrs::p2p] loading 10 blocks
[2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed
[2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected
[2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send
[2024-08-18T10:06:48.067Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin
Error: electrs failed

Caused by:
0: sync failed
1: failed to get block 00000000000000000785bdd27a61fdd9c82b033871cf544b2a77cda0d0ff321e
2: receiving on an empty and disconnected channel

and electrs never restart until restart bitcoind
I've tried to delete /opt/data/electrs/.electrs/db/bitcoin content without lucky

[asyscom]

Check bitcoind's doc.

The last attempt was to uninstall and reinstall Electrs, but it still doesn't start. I have to restart Bitcoin Core to get it running again, but after just a few minutes of syncing, the problem reoccurs. Any suggestions?

[antonilol]

How is ram, cpu and disk usage just before the error occurs?

[asyscom]

How is ram, cpu and disk usage just before the error occurs?

Hello, I've 27gb of ram, 500gb of free disk on total of 2tb and 8 cpu

[Kixunil]

Any logs in bitcoind around that time?

[antonilol]

Hello, I've 27gb of ram, 500gb of free disk on total of 2tb and 8 cpu

Assuming that's 27 GB free (not total) that should be more than enough, I have had issues with bitcoind rpc timing out on close to 100% memory usage or prolonged 100% cpu usage.
500 GB free disk is also more than enough. My electrs mainnet database is currently around 48 GB.