Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto sync score-client to operate on same storage profile as score-server #378

Open
UmmulkiramR opened this issue Nov 13, 2023 · 3 comments · Fixed by #382
Open

Auto sync score-client to operate on same storage profile as score-server #378

UmmulkiramR opened this issue Nov 13, 2023 · 3 comments · Fixed by #382
Assignees
Labels
new-feature Request is a new feature

Comments

@UmmulkiramR
Copy link
Contributor

UmmulkiramR commented Nov 13, 2023

At the moment, score-client and score-server each operate on a storage profile (azure or s3), and they need to be synced to make sure the correct object storage system is called. If not in sync the uploads/downloads will fail. Users, however, should not have to worry about operating score-client on the right profile. Hence, this change is proposed where score-client will auto-sync itself with the profile of score-server.

As part of this change,

  1. an endpoint (eg. _getStorageProfiles_) will be added to score-server that returns the active profiles it is operating on.
  2. when executing a command, score-client will first make a call to this _getStorageProfiles_ endpoint to get the active profiles.
  3. based on the response of this API call score-client will switch its profile to be the same as score-server.
  4. in order to maintain compatibility with older versions of score-server, in the absence of the _getStorageProfiles_ endpoint score-client will use the existing logic to select the right profile which is defaulting to S3 storage.
  5. the list of storage profiles currently available are azure and s3
  6. the storage profile values score-server is expected to respond with are "azure" for azure or "amazon" for s3.
score_client_server_comm
@UmmulkiramR UmmulkiramR added the new-feature Request is a new feature label Nov 13, 2023
@UmmulkiramR UmmulkiramR self-assigned this Nov 13, 2023
UmmulkiramR added a commit that referenced this issue Nov 24, 2023
* added server endpoint to get profiles

* change to switch score-client storage implementations

* cleared azure and s3 related entries app.yml

* some bug fixes and enhancements

* updated comment

* added test profile.

* added test profile.

* refactored code based on review comments - BaseController now returns a single profile value. The actual profile name and the profile value returned by the api are now different.

* debug logging removed

* replaced profile value

* updated readme

* updated readme

* added a test profile

* review changes
- Storage profile values now come from an enum in score-core
- test configuration created to mock storage profile bean
- users will be able to provide a default profile value when working with old score-server instances

* users will be able to provide a default profile value when working with old score-server instances

* config change

* added a test config in score server

---------

Co-authored-by: UmmulkiramR <[email protected]>
@UmmulkiramR
Copy link
Contributor Author

merged to develop

@UmmulkiramR UmmulkiramR linked a pull request Nov 24, 2023 that will close this issue
UmmulkiramR added a commit that referenced this issue Nov 27, 2023
* Readme updated

* mergeback for 5.9.1-SNAPSHOT

* updated readme copy

* updated readme copy

* Update README.md

Co-authored-by: Jon Eubank <[email protected]>

* CORS configurations for score server allowing multiple origins at once - #367

* Organizing score-server application.yml

* docker JRE ireplacing JRE alpine image to support multiple architecturesmage multi arch supported

* latest ubuntu LTS 22.04 score client

* using JDK image as builder

* update docker dind (#376)

from docker image docker:18.06-dind to docker:20.10-dind

* fix docker dind TLS (#379)

* increase Jenkins timeout (#380)

from 30 to 45 mins

* Fix/jenkins extend timeout (#381)

* increase Jenkins timeout

from 30 to 45 mins

* increase Jenkins timeout

from 45 to 60 mins

* increase jenkins timeout

* update DeployWithHelm job name in Jenkinsfile (#383)

* fix for issue #385 (#386)

Co-authored-by: UmmulkiramR <[email protected]>

* Auto sync storage profiles #378 (#382)

* added server endpoint to get profiles

* change to switch score-client storage implementations

* cleared azure and s3 related entries app.yml

* some bug fixes and enhancements

* updated comment

* added test profile.

* added test profile.

* refactored code based on review comments - BaseController now returns a single profile value. The actual profile name and the profile value returned by the api are now different.

* debug logging removed

* replaced profile value

* updated readme

* updated readme

* added a test profile

* review changes
- Storage profile values now come from an enum in score-core
- test configuration created to mock storage profile bean
- users will be able to provide a default profile value when working with old score-server instances

* users will be able to provide a default profile value when working with old score-server instances

* config change

* added a test config in score server

---------

Co-authored-by: UmmulkiramR <[email protected]>

* rc release. Includes
 - auto sync storage profiles - #387
 - azure download fails - #385
 - fix in Jenkins to build the Score docker image to run on amd/arm architectures and updating score-client image to use latest LTS Ubuntu- #374 

---------

Co-authored-by: Mitchell Shiell <[email protected]>
Co-authored-by: dahiyaAD <[email protected]>
Co-authored-by: dahiyaAD <[email protected]>
Co-authored-by: Jon Eubank <[email protected]>
Co-authored-by: Leonardo Rivera <[email protected]>
Co-authored-by: UmmulkiramR <[email protected]>
@lindaxiang
Copy link

lindaxiang commented Nov 28, 2023

@UmmulkiramR
I tested both data submission and download in cumulus rdpc-qa for small datasets using score version 5.10.0 without providing profile, and both succeeded. I used subworkflow in argo-modules to test both:

  • submission
    nextflow run ./tests/subworkflows/icgc-argo-workflows/song_score_upload -entry test_song_score_upload_user -profile docker -c ./tests/config/nextflow.config --api_token *** --study_id TEST-PR --payload ./tests/data/qa/payload.sequencing_experiment.json --upload_files "./tests/data/qa/test_rg_3*.v2.bam" --song_url "https://submission-song.rdpc-qa.cumulus.genomeinformatics.org" --score_url "https://submission-score.rdpc-qa.cumulus.genomeinformatics.org" --song_container_version "5.0.2" --score_container_version "5.10.0" --song_container "ghcr.io/overture-stack/song-client" --score_container "ghcr.io/overture-stack/score"
  • download
    nextflow run ./tests/subworkflows/icgc-argo-workflows/song_score_download -entry test_song_score_download_user -profile docker -c ./tests/config/nextflow.config --api_token *** --study_id TEST-QA --analysis_id 2c392294-7a1c-4de8-b922-947a1c1de803 --song_url "https://song.rdpc-qa.cumulus.genomeinformatics.org" --score_url "https://score.rdpc-qa.cumulus.genomeinformatics.org" --song_container_version "5.0.2" --score_container_version "5.10.0" --song_container "ghcr.io/overture-stack/song-client" --score_container "ghcr.io/overture-stack/score"

@UmmulkiramR
Copy link
Contributor Author

Deployed on prod

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-feature Request is a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants