Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Segmentation Fault Error at indexing blocks [180001..182000] #1061

Open
adriankae opened this issue Jul 12, 2024 · 19 comments
Open

Bug: Segmentation Fault Error at indexing blocks [180001..182000] #1061

adriankae opened this issue Jul 12, 2024 · 19 comments
Labels
bug Something isn't working

Comments

@adriankae
Copy link

adriankae commented Jul 12, 2024

Describe the bug
Everytime I start electrs I get a seg fault error at indexing the blocks. Specifically at blocks [180001..182000]. It's always in the same spot.
Any help would be appreciated.

Electrs version
v.0.10.5

To Reproduce
Steps to reproduce the behavior:

  1. Configure and start electrs
  2. Wait
  3. See error

Expected behavior
Indexing throws no errors.

Configuration

electrs.toml Didn't find it in the directory.
[2024-07-12T16:42:00.610Z INFO  electrs::index] indexing 2000 blocks: [180001..182000]
Segmentation fault (core dumped)

running with gdb:


Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff58006c0 (LWP 3120)]
0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()

Environment variables: ELECTRS_X=Y;...
Arguments: --foo

System running electrs

  • Deployment method: manual from source
  • OS name and version: Ubuntu Server 24.04 LTS

Electrum client
Client name (if not upstream desktop Electrum) and version: no electrum installed

Additional context
n/a

@adriankae adriankae added the bug Something isn't working label Jul 12, 2024
@adriankae
Copy link
Author

error running with gdb:

[2024-07-12T16:55:11.385Z INFO electrs::index] indexing 2000 blocks: [180001..182000]

Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff58006c0 (LWP 3120)]
0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()

@Kixunil
Copy link
Contributor

Kixunil commented Jul 12, 2024

Did you link dynamically or statically?

@adriankae
Copy link
Author

Did you link dynamically or statically?

I linked statically.

@Kixunil
Copy link
Contributor

Kixunil commented Jul 12, 2024

That's really weird, probably has something to do with C++ compilation. Makes me rethink my previous claim that maybe pure Rust impl isn't that interesting.

@adriankae
Copy link
Author

When I try to link dynamically i get these errors:

error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
   --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:532:18
    |
532 |             ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache);
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
    |
   ::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
    |
3   | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
    |       ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here

error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
   --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:553:18
    |
553 |             ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache.0.inner);
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
    |
   ::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
    |
3   | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
    |       ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here

error[E0061]: this function takes 9 arguments but 5 arguments were supplied
    --> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:2525:13
     |
2525 |               ffi::rocksdb_options_set_plain_table_factory(
     |  _____________^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^-
2526 | |                 self.inner,
2527 | |                 options.user_key_length,
2528 | |                 options.bloom_bits_per_key,
2529 | |                 options.hash_table_ratio,
2530 | |                 options.index_sparseness,
2531 | |             );
     | |_____________- multiple arguments are missing
     |
note: function defined here
    --> /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:88313
     |
3    | ...size) ; } extern "C" { pub fn rocksdb_options_set_plain_table_factory (arg1 : * mut rocksdb_options...
     |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: provide the arguments
     |
2525 |             ffi::rocksdb_options_set_plain_table_factory(self.inner, options.user_key_length, options.bloom_bits_per_key, options.hash_table_ratio, options.index_sparseness, /* usize */, /* i8 */, /* u8 */, /* u8 */);
     |                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some errors have detailed explanations: E0061, E0425.
For more information about an error, try `rustc --explain E0061`.
error: could not compile `electrs-rocksdb` (lib) due to 3 previous errors

@Kixunil
Copy link
Contributor

Kixunil commented Jul 12, 2024

Yes, Ubuntu has a different rocksdb version than Debian. I wish they didn't break the API so much.

@adriankae
Copy link
Author

Is there anything I could do?
I'm pretty much out of ideas...

@Kixunil
Copy link
Contributor

Kixunil commented Jul 12, 2024

I don't have many ideas either. Maybe try switching compiler to clang? (If you used g++, or vice-versa).

@romanz
Copy link
Owner

romanz commented Aug 12, 2024

@adriankae Does the error reproduce if the index DB is rebuilt from scratch?

@bkjaerland
Copy link

I have the exact same issue. For me it did not help to rebuild the DB from scratch.

@bkjaerland
Copy link

Found the issue for me, I was missing the build-essential package. After installing it and rebuilding electrs it works like a charm.

@Kixunil
Copy link
Contributor

Kixunil commented Aug 25, 2024

How come it even compiled without it?! That's super weird.

@antonilol
Copy link
Contributor

Is this a rocksdb issue? I think somewhere a compile time error should be introduced.

@xjmzx
Copy link

xjmzx commented Sep 1, 2024

Confirming the same issue (at same block height) and the same fix build-essential was not installed.

OS: Ubuntu 24.04.1 LTS x86_64
Kernel: 6.8.0-41-generic

@jozbone
Copy link

jozbone commented Sep 7, 2024

Just confirming as well that I was seeing the same issue at the same block height on Ubuntu 24.04. Installed "build-essential" and rebuilt electrs (static linking). After that I was able to run it and get past the previously failing block height point.

@Kixunil
Copy link
Contributor

Kixunil commented Sep 7, 2024

I don't think we need more reports, but new people coming to the issue can just thumbs up this comment: #1061 (comment)

What would be really helpful is to abort compilation if this is happening but that requires some serious digging and will likely need an issue filed against the cc crate (but only once it's confirmed to belong there - please don't pester them about it now).

If someone figures out more information about this that would explain why this happens that'd be great. Sadly I don't have that much time for this kind of thing lately.

@romanz
Copy link
Owner

romanz commented Oct 15, 2024

On Ubuntu 24.04 installing clang results in Clang 18:

$ c++ --version
Ubuntu clang version 18.1.3 (1ubuntu1)
...

If I then install build-essential, GCC 13 becomes the default:

$ c++ --version
c++ (Ubuntu 13.2.0-23ubuntu4) 13.2.0
...

Also, mempool/electrs#106 also stopped using Clang 18.

@romanz
Copy link
Owner

romanz commented Oct 29, 2024

Reproduced using a Ubuntu 24.04 Docker container:

$ gdb --args /usr/bin/electrs --log-filters INFO
(gdb) r
...
[2024-10-29T14:15:03.607Z INFO  electrs::chain] chain updated: tip=00000000000004ff83b6c10460b239ef4a6aa320e5fffd6c7bcedefa8c78593c, height=180000
[2024-10-29T14:15:03.608Z INFO  electrs::index] indexing 2000 blocks: [180001..182000]

Thread 10 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7bb3c1e006c0 (LWP 1032)]
0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
(gdb) bt
#0  0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
#1  0x000064b8f5f8074c in rocksdb::CompressionContextCache::GetCachedZSTDUncompressData() ()
#2  0x000064b8f5f177c7 in rocksdb::UncompressionContext::UncompressionContext(rocksdb::CompressionType) ()
#3  0x000064b8f5f4c054 in rocksdb::BlockFetcher::ReadBlockContents() ()
#4  0x000064b8f5f1f413 in rocksdb::Status rocksdb::(anonymous namespace)::ReadBlockFromFile<rocksdb::Block>(rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, std::unique_ptr<rocksdb::Block, std::default_delete<rocksdb::Block> >*, rocksdb::ImmutableOptions const&, bool, bool, rocksdb::BlockType, rocksdb::UncompressionDict const&, rocksdb::PersistentCacheOptions const&, unsigned long, rocksdb::MemoryAllocator*, bool, bool, rocksdb::FilterPolicy const*, bool) ()
#5  0x000064b8f5f164a5 in rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::Block>(rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::CachableEntry<rocksdb::Block>*, rocksdb::BlockType, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, bool, bool, bool, bool) const ()
#6  0x000064b8f5f431f3 in rocksdb::BlockBasedTable::IndexReaderCommon::ReadIndexBlock(rocksdb::BlockBasedTable const*, rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocksdb::CachableEntry<rocksdb::Block>*) ()
#7  0x000064b8f6035b8e in rocksdb::BinarySearchIndexReader::Create(rocksdb::BlockBasedTable const*, rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#8  0x000064b8f5f25d54 in rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#9  0x000064b8f5f250b6 in rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, rocksdb::BlockBasedTable*, bool, rocksdb::BlockBasedTableOptions const&, int, unsigned long, unsigned long, rocksdb::BlockCacheLookupContext*) ()
#10 0x000064b8f5f22706 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::CacheReservationManager>, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::array<unsigned long, 2ul>) ()
#11 0x000064b8f5f0ca46 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const ()
#12 0x000064b8f5e2c61c in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl*, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#13 0x000064b8f5e2b497 in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Cache::Handle**, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#14 0x000064b8f5e2cb10 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::shared_ptr<rocksdb::SliceTransform const> const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocksdb::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool, rocksdb::TruncatedRangeDelIterator**) ()
#15 0x000064b8f5fbc8a8 in rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) ()
#16 0x000064b8f5de44ab in rocksdb::FlushJob::WriteLevel0Table() ()
#17 0x000064b8f5de040d in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) ()
#18 0x000064b8f5d71fe7 in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::vector<unsigned long, std::allocator<unsigned long> >&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#19 0x000064b8f5d73238 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#20 0x000064b8f5d85428 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) ()
#21 0x000064b8f5d83b88 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) ()
#22 0x000064b8f5f8e4f5 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#23 0x000064b8f5f8e759 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#24 0x00007bb3c488fbb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#25 0x00007bb3c4519a94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#26 0x00007bb3c45a6a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

with the following diff:

diff --git a/Dockerfile b/Dockerfile
index c8ec506e..a6f1dba8 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,19 +2,18 @@
 # The maintainers of electrs are not deeply familiar with Docker, so you should DYOR.
 # If you are not familiar with Docker either it's probably be safer to NOT use it.
 
-FROM debian:bookworm-slim AS base
+FROM ubuntu:24.04 AS base
 RUN apt-get update -qqy
-RUN apt-get install -qqy librocksdb-dev curl
+RUN apt-get install -qqy curl gdb bash
 
 ### Electrum Rust Server ###
 FROM base AS electrs-build
RUN apt-get install -qqy cargo clang cmake
 
 # Install electrs
 WORKDIR /build/electrs
 COPY . .
-ENV ROCKSDB_INCLUDE_DIR=/usr/include
-ENV ROCKSDB_LIB_DIR=/usr/lib
 RUN cargo install --locked --path .
 
 FROM base AS result

@schmitmd
Copy link

schmitmd commented Nov 17, 2024

I can confirm that on FreeBSD, installing llvm17 and recompiling electrs with the following set of variables does indeed fix this.

CC=/usr/local/bin/clang17
CXX=/usr/local/bin/clang++17
CPP=/usr/local/bin/clang-cpp17
RUSTFLAGS="-C linker=clang17"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants