Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert memtable rotation to a channel message #78

Open
marvin-j97 opened this issue Sep 21, 2024 · 0 comments
Open

Convert memtable rotation to a channel message #78

marvin-j97 opened this issue Sep 21, 2024 · 0 comments
Labels
enhancement New feature or request performance

Comments

@marvin-j97
Copy link
Collaborator

marvin-j97 commented Sep 21, 2024

Currently the memtable rotation happens in the write hot path when the memtable size threshold is reached:

fjall/src/partition/mod.rs

Lines 747 to 752 in 9395ba1

pub(crate) fn check_memtable_overflow(&self, size: u32) -> crate::Result<()> {
if size > self.config.max_memtable_size {
self.rotate_memtable()?;
self.check_journal_size();
self.check_write_halt();
}

The performance hit mostly comes from the journal rotation which needs to fsync 2x (?) to disk.
It would be better for performance to send a message via channel, announcing that the partition needs to be flushed, nullifying rotation costs inside the write hot path, which should give quite a performance boost for write heavy applications (currently it is slightly slower than Rocks).

The messages should be picked up by a worker that rotates the partition asynchronously, essentially performing the Partition::rotate_memtable function:

pub fn rotate_memtable(&self) -> crate::Result<bool> {

@marvin-j97 marvin-j97 added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed performance labels Sep 21, 2024
@marvin-j97 marvin-j97 pinned this issue Sep 21, 2024
@marvin-j97 marvin-j97 transferred this issue from fjall-rs/lsm-tree Sep 22, 2024
@marvin-j97 marvin-j97 pinned this issue Oct 21, 2024
@marvin-j97 marvin-j97 unpinned this issue Oct 21, 2024
@marvin-j97 marvin-j97 removed help wanted Extra attention is needed good first issue Good for newcomers labels Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
None yet
Development

No branches or pull requests

1 participant