Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update index workflow to pass a subpath to extract the kraken2 DB #91

Open
harmonbhasin opened this issue Nov 13, 2024 · 1 comment
Open
Assignees
Labels
done Issues that have been addressed in dev branch, but not reflected in master branch enhancement New feature or request priority_3

Comments

@harmonbhasin
Copy link
Collaborator

Due to new updates in permissions made by the krakenDB hosts, the kraken DB can no longer be pulled by passing in the s3 path to nextflow as they've changed permissions on the repo to only allow for unsigned requests (this is an issue as we need to be logged in for aws batch to store files in our private s3 buckets). Currently, Will has downloaded the kraken db to an NAO s3 bucket, however it's large (~70 GB), so we probably don't want to keep downloading this and keeping it.

A better solution would be to pull the KrakenDB during the index workflow by only passing in the subpath, then creating a process that downloads the krakenDB by calling aws s3 cp --no-sign-request followed by the path.

@harmonbhasin harmonbhasin added enhancement New feature or request priority_3 labels Nov 13, 2024
@harmonbhasin
Copy link
Collaborator Author

Turns out if you use "https://" instead of "s3://" it works perfectly fine.

@harmonbhasin harmonbhasin added the done Issues that have been addressed in dev branch, but not reflected in master branch label Nov 18, 2024
@harmonbhasin harmonbhasin self-assigned this Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
done Issues that have been addressed in dev branch, but not reflected in master branch enhancement New feature or request priority_3
Projects
None yet
Development

No branches or pull requests

1 participant