Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change quickstart to enable neural and hybrid search #127

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

wrigleyDan
Copy link

This PR makes Chorus neural and hybrid search ready by default.

This includes:

  • Prepare OpenSearch to run ML models
  • Generate embeddings during index time
  • Add pipelines for neural and hybrid search

@wrigleyDan wrigleyDan marked this pull request as draft November 27, 2024 10:32
… when switching between keyword, neural, hybrid. Add search pipelines to quickstart.sh for neural and hybrid
@wrigleyDan
Copy link
Author

waiting for PR #119 to be merged into main. Then main needs to be merged into this branch and it's ready for review.

@wrigleyDan wrigleyDan marked this pull request as ready for review November 28, 2024 12:52
}')

# Extract the model_group_id from the JSON response
model_group_id=$(echo "$response" | jq -r '.model_group_id')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow! Good job.

echo -e "${MAJOR}Registering a model in the model group.${RESET}"
response=$(curl -s -X POST "http://localhost:9200/_plugins/_ml/models/_register" \
-H 'Content-Type: application/json' \
--data-binary "{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think we need all the \ characters.. We don't have that in other examples... where the --data-binary uses a '

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at first, collapsing down to a single index didn't sit right, and yet, now it does, beuae hybrid is just "search". we don't need to segrategate our indexes by how we parse and index the data!

@epugh
Copy link
Member

epugh commented Dec 2, 2024

okay, we definitly need to make it run faster... Maybe we have an option "--full-data-set" and other wise only load two of the files??? Whyd oesn't it index faster, is it all the embeddings being generated?

@epugh
Copy link
Member

epugh commented Dec 2, 2024

I tried to clean things up, but started getting errors in running opensearch, not sure if it's the patch:

patch.patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make Chorus "neural"- and "hybrid-search-ready" with the quickstart.sh script
2 participants