Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Introduce scalable query nodes #991

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

parmesant
Copy link
Contributor

This PR introduces distributed, scalable querying in a stateless manner. The idea is that query nodes should be able to scale up and down like ingestor nodes. The only difference being that since these nodes are also writing data to S3, we have to have one leader to coordinate API calls and writing operations.

Description


This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

TODO:

  • Thorough testing
  • Migration flow validation
  • Ingestor stream creation
  • Pulling data from ingestor staging in a query

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant