Clickhouse api #11224

alisman · 2024-11-22T20:24:16Z

Fix # (see https://help.github.com/en/articles/closing-issues-using-keywords)

Describe changes proposed in this pull request:

a
b

Checks

The commit log is comprehensible. It follows 7 rules of great commit messages. We can fix this during merge by using a squash+merge if necessary
Has tests or has a separate issue that describes the types of test that should be created. If no test is included it should explicitly be mentioned in the PR why there is no test.
Is this PR adding logic based on one or more clinical attributes? If yes, please make sure validation for this attribute is also present in the data validation / data loading layers (in backend repo) and documented in File-Formats Clinical data section!
Make sure your PR has one of the labels defined in https://github.com/cBioPortal/cbioportal/blob/master/.github/release-drafter.yml

Any screenshots or GIFs?

If this is a new visual feature please add a before/after screenshot or gif
here with e.g. Giphy CAPTURE or Peek

Notify reviewers

Read our Pull request merging
policy. It can help to figure out who worked on the
file before you. Please use git blame <filename> to determine that
and notify them either through slack or by assigning them as a reviewer on the PR

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

* ✅ Add Unit test for StudyViewMapper Clickhouse * ✅ Update db props to include mysql and clickhouse datasources to fix tests * Address comments * Rename package to clickhouse * Update to static final * Use bean name instead of qualifier

* Create new wide table sql file and rename package * Remove genomic_event view * Add AlterationFilter to mutated_genes endpoint * Add AlterationFilter to mutated-genes endpoint * Fix unit test * Fix sonar issues * Add test for mutation types and status * remove unused imports

* add missing poc clinical data binning function

* Add sample_mv materialized view and use it in mappers

* Add Support for TotalProfiledCase Counts for Mutated-genes endpoint. * Create sql files to create new tables * Add unit test for totalProfiledCount * Add matching gene panel ids * Add TotalProfiledCountsWithoutPanelData * Add profileCount for genes without gene panel data * Add Comments for SQL * Update matching Gene Panel Ids * Clean up code * Fix test * Add query to get correct Gene Panels * Fix unit test * Add comments

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

* Implement molecular profile count endpoint using Clickhouse * Cleanup

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Co-authored-by: Bryan Lai <[email protected]>

* use clinical_data_derived instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * use clinical_attribute_meta instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * remove unused clinical data count methods and SQL * fix numericalClinicalDataCountFilter * Move CategoricalClinicalAttributeFilter to repository * remove unused columns * Add override to methods --------- Co-authored-by: haynescd <[email protected]>

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

* simplify clinical data binning related SQL * fix numericalClinicalDataCountFilter

…ase64) (#10898)

* Add Clinical Events Timeline counts endpoint * Add comments for StudyViewFilter

* Fix alteration count services * Only add genes that are protein coding to gene_panel_to_gene table for WES * fix test * fix sonar issues * Add documentation for filtering out WES

…n assume total Profile Count

…tion

…lter-param Fix error with missing study view filter param in clickhouse method caching condition

…dy view (#11185) * Fix counting of clinical data na when filtered out * Add caching and cleanup * Fix Sonar issues

* add api tests for circleci

This reverts commit 80ef8c9.

* add tests for profiled counts * fix profiled counts assertions --------- Co-authored-by: Bryan Lai <[email protected]>

#11209) * separate clickhouse tests into different files * change name of cna counts test file --------- Co-authored-by: Bryan Lai <[email protected]>

…inical data counts endpoint (#11192) Co-authored-by: alisman <[email protected]>

Use PR-specific image for circleci api tests

Require manual approval from admins for circleci api tests.

alisman · 2024-11-22T20:25:36Z

src/main/resources/org/cbioportal/persistence/mybatis/ClinicalEventMapper.xml

@@ -181,7 +181,7 @@
                </if>
            </if>
        </where>
-        Group by clinical_event.EVENT_TYPE, patient.STABLE_ID
+        Group by clinical_event.EVENT_TYPE, patient.INTERNAL_ID


@haynescd this is one fix i did. i think we can keep it. the stable id is not unique across study

alisman · 2024-11-22T21:10:11Z

src/main/java/org/cbioportal/service/impl/AlterationCountServiceImpl.java

@@ -223,6 +263,197 @@ public Pair<List<CopyNumberCountByGene>, Long> getPatientCnaGeneCounts(List<Mole
        );
    }

+    @Override


@haynescd what's all this stuff?

alisman · 2024-11-22T21:16:33Z

src/main/resources/db-scripts/clickhouse/materialized_views.sql

@@ -0,0 +1,32 @@
+DROP TABLE IF EXISTS sample_list_columnstore;


@haynescd we can kill this file, right?

alisman · 2024-11-22T21:18:22Z

src/main/resources/org/cbioportal/persistence/mybatis/GenePanelMapper.xml

@@ -121,6 +121,7 @@
        <include refid="selectGenePanelData"/>
        <include refid="fromGenePanelData"/>
        WHERE
+        SAMPLE_ID IS NOT NULL AND 


@haynescd this is one we might want to undo because it changes profile counts. this is what allows the system to recover from incomplete sample_profile table issue.

alisman · 2024-11-22T21:19:11Z

src/main/resources/templates/index.html

@@ -32,7 +32,7 @@
        window.netlify = localStorage.netlify;

        if (window.localdev || window.localdist) {
-            window.frontendConfig.frontendUrl = "//localhost:3000/"
+            window.frontendConfig.frontendUrl = "https://localhost:3000/"


might possible break localdb tests

Co-authored-by: Bryan Lai <[email protected]>

haynescd and others added 30 commits April 8, 2024 12:35

➕ Add infra to support multi db and Columnar DataSource Properties

db9e6ea

Add Columnar support for Mybatis

7d8b9f7

Add Columnar Study View Controller /api/column-store

8b07d3f

Feature/ch/update poc base branch (#10776)

63dde5f

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

use regular views instead of materialized views (#10793)

77d07e0

Add missing POC clinical data binning function (#10778)

d24648a

* add missing poc clinical data binning function

Create sample materialized view and use it (#10822)

e37fa86

* Add sample_mv materialized view and use it in mappers

RFC80:density plot endpoint(#10837)

1d1606b

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

Fix clinical data counts query/filter using new derived table (#10833)

464f442

Molecular-profile-counts service implementation (#10840)

6998fc0

* Implement molecular profile count endpoint using Clickhouse * Cleanup

Fix clinical data scatterplot (point at new derived cd table) (#10846)

0b8c89e

Rfc80/add cna genes endpoint (#10845)

bb5456a

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

✨ Add StructuralVariant-genes endpoint (#10854)

505847b

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Implement study view sample-lists-counts endpoint in Clickhouse (#10849)

74b735d

Co-authored-by: Bryan Lai <[email protected]>

Fix class import (#10856)

110236b

Rfc80/fix patient level filtering for alteration counts endpoints (#1…

14b7105

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

🐛 Fix totalCount calculation when WES not available (#10858)

2b87a26

Consolidate SQL files used to create CH Tables (#10867)

79d36e7

Demo rfc80 poc violin plot (#10863)

2e02f21

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

replace sample_mv references with sample_derived (#10880)

0fc686b

🔧 Update Sonar Cloud Token (#10888)

f602587

Simplify clinical data binning related SQL (#10823)

0dfb346

* simplify clinical data binning related SQL * fix numericalClinicalDataCountFilter

Fix patient_derived table population (patient stable id was getting b…

6df137f

…ase64) (#10898)

Always load frontend over https (localdev mode) (#10901)

3455f97

Rfc80/clinical event timeline endpoint (#10899)

77bf77f

* Add Clinical Events Timeline counts endpoint * Add comments for StudyViewFilter

Fix alteration count services (#10911)

18a5ee7

* Fix alteration count services * Only add genes that are protein coding to gene_panel_to_gene table for WES * fix test * fix sonar issues * Add documentation for filtering out WES

haynescd and others added 24 commits November 7, 2024 14:29

Update Profiled counts to use WES count when gene panel is not available

ad6279f

Update to handle use case when sampleProfileWithoutGenePanel is 0 the…

d1f3893

…n assume total Profile Count

add custom driver filtering unit tests

80ef8c9

add missing study filter param to clickhouse method for caching condi…

fc9ef56

…tion

Merge pull request #11154 from gblaih/demo-rfc80-poc-missing-study-fi…

122f4ae

…lter-param Fix error with missing study view filter param in clickhouse method caching condition

CNA profile counts should only count samples once

49d6952

Add back NA counts when corresponding samples are filtered out of stu…

6e4d297

…dy view (#11185) * Fix counting of clinical data na when filtered out * Add caching and cleanup * Fix Sonar issues

Update generic assay table schema (#11189)

aa7fffe

Do not used prepared statements for the studyid lists (#11191)

41b30f3

Api Tests Config for CircleCI (#11188)

b08c011

* add api tests for circleci

Revert "add custom driver filtering unit tests" (#11203)

172ccd3

This reverts commit 80ef8c9.

Update Total profile Count to filter on genetic profile (#11198)

6221e88

Add tests for profiled counts (#11190)

2b3b6e0

* add tests for profiled counts * fix profiled counts assertions --------- Co-authored-by: Bryan Lai <[email protected]>

Remove test failing due to legacy miscount (#11205)

a56a698

cast all data filter start and end values to float (#11207)

79dc173

Split clickhouse integration tests into separate files for each method (

1361bf6

#11209) * separate clickhouse tests into different files * change name of cna counts test file --------- Co-authored-by: Bryan Lai <[email protected]>

Adding option for extracting study ids from sample identifiers for cl…

b94e39e

…inical data counts endpoint (#11192) Co-authored-by: alisman <[email protected]>

Add missing DROP TABLE statements (#11211)

7295a27

Use PR-specific image for circleci api tests (#11217)

792dd30

Use PR-specific image for circleci api tests

Pretty print portal info in api-test logs

170d962

Add a frontend prop for clickhouse mode (#11219)

aff5156

require approval for restricted tests

6a1508a

rename approval job for circleci tests

ad57e8a

Merge pull request #11223 from cBioPortal/feat/circleci-approval

c039cad

Require manual approval from admins for circleci api tests.

alisman commented Nov 22, 2024

View reviewed changes

add integration tests for cna-genes and sv-genes (#11221)

4d4b302

Co-authored-by: Bryan Lai <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clickhouse api #11224

Clickhouse api #11224

alisman commented Nov 22, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

		@@ -0,0 +1,32 @@
		DROP TABLE IF EXISTS sample_list_columnstore;

Clickhouse api #11224

Are you sure you want to change the base?

Clickhouse api #11224

Conversation

alisman commented Nov 22, 2024

Checks

Any screenshots or GIFs?

Notify reviewers

alisman Nov 22, 2024

Choose a reason for hiding this comment

alisman Nov 22, 2024

Choose a reason for hiding this comment

alisman Nov 22, 2024

Choose a reason for hiding this comment

alisman Nov 22, 2024

Choose a reason for hiding this comment

alisman Nov 22, 2024

Choose a reason for hiding this comment