Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternative sources of value tables #4376

Open
ymarcon opened this issue Nov 29, 2022 · 0 comments
Open

Alternative sources of value tables #4376

ymarcon opened this issue Nov 29, 2022 · 0 comments
Assignees
@ymarcon
Labels
new feature

Comments

@ymarcon
Copy link
Member

ymarcon commented Nov 29, 2022 •
edited
Loading

Currently Mica depends on Opal for getting the data dictionaries and summary statistics. The proposal is to make use the Magma abstraction to access ValueTables from other Datasource than the RestDatasource. For instance, the ExcelDatasource could be used to retrieve data dictionaries from a file stored in Mica.

The work would then consist of generalizing the StudyTable attached to a Dataset so that it is not limited to the "project" and "table" properties which are Opal specific. A URN could be used instead, to identify the underlying ValueTable. Its location would be resolved in the context of the Study/Dataset: for instance when a specific Opal server URL is defined for the Study, the table's location can be resolved by combining this Opal server URL and the StudyTable's URN.

This also opens the possibility to have a "study-table-source" plugin type, to support future alternate sources of value tables.

These changes would also affect the variable summary and contingency retrieval, as the statistics may be differently supported by the underlying Datasource: for instance, when using the ExcelDatasource, the summary statistics could be extracted statically from the variable attributes and the contingency feature would not be available.

Affected components:

  • Dataset / StudyTable / DatasetVariable classes
  • Dataset services
  • Variable indexing and caching
  • Dataset resources
  • Dataset administration UI
@ymarcon ymarcon self-assigned this Nov 29, 2022
ymarcon added a commit that referenced this issue Nov 29, 2022
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
643c696 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
@ymarcon ymarcon moved this to In Progress in NFDI4Health Dec 11, 2022
@ymarcon ymarcon mentioned this issue Dec 11, 2022
Open
ymarcon added a commit to obiba/mica-search-es that referenced this issue Dec 14, 2022
@ymarcon
Properties 'project' and 'table' replaced by 'source' to support diff…
f8cefca 
…erent variable origins (not necessarily opal) obiba/mica2#4376
ymarcon added a commit that referenced this issue Jan 11, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
34e8906 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 19, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
b4bd97d 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 19, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
c151256 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
62a196c 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
fd36162 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
71d3e21 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 24, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
1a6ebcc 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 26, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
1dd8a53 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 27, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
8b3b889 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 30, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
d2c8906 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
53e6922 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
dcea4db 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
afa10f1 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Feb 1, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
2bba3a1 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
ymarcon added a commit that referenced this issue Feb 1, 2023
@ymarcon
#4376 WIP. StudyTableSource interface added. New property sourceURN r…
af4f178 
…eplaces opal project/table in dataset's study tables and variables. Requires ES index config update.
@ymarcon ymarcon mentioned this issue Feb 1, 2023
Merged
@ymarcon ymarcon moved this from In Progress to Done in NFDI4Health Feb 1, 2023
meek0 pushed a commit that referenced this issue Feb 1, 2023
@ymarcon
Mica without Opal (#4393)
2101d87 
* #4376 WIP. StudyTableSource interface added. New property sourceURN replaces opal project/table in dataset's study tables and variables. Requires ES index config update.

* sourceURN property/getter/setter renamed to source

* source package added, with unit tests

* Excel table source added, reads excel from Mica's file system

* Edit/create study tables with opal or file source. WIP.

* File path can be copied. Study table's file source can be relative to the dataset's folder

* Study table context added. Opal dtos (aggregations) are not exposed in the source plugin interface.

* Deferred init of excel datasource. Less verbose log messages when study table source operation is not supported

* SPI mica-source testing

* Deferred attachment stream extraction

* Handle disposable table sources

* Mica source plugin could be an Initialisable

* #4381 Variable taxonomies service added to support other taxonomies providers, in addition to the opal one

* mica-source plugin type renamed to mica-tables. Ne plugin type: mica-taxonomies for loading taxonomies provider services.

* Taxonomy of taxonomies is configurable. Added attribute to have a taxonomy declared but not visible.

* Variable taxonomies can be read from local folder: MICA_HOME/conf/taxonomies/variable

* AbstractTaxonomiesProviderService added

* Caching of taxonomies refactored, not functional yet

* Variable taxonomies are cached

* Fix dataset taxonomy init

* Opal taxonomies cache replaced by generic variable taxonomies cache. Variable taxonomies are decorated with variable attributes

* Code cleaning: Opal service helper merged into opal service

* classifications administration

* make sure variable taxo title/description in the meta taxo matches the one of the original taxo

* hide instead of remove a target vocabulary if type is disabled

* Plugins REST API added

* Plugin REST API added

* Service Plugin REST API added

* Update the search plugin in the upgrade procedure

* ng-obiba-mica version

* No server error when opal projects cannot be retrieved

* Fix some wording and showVariableStatistics template setting added

* Fix some wording and showVariableStatistics template setting added

* Fix harmonized variable id encoding for url

* Fix harmonized variable id encoding for url for reverse proxy

* NoSuchTableSourceException added with their web service mapper

* ng-obiba-mica latest release

* prepare for upgrade to 5.2

* StudyTableSource does not need to return the table name, this is implementation specific

* Modifying the meta txonomy requires admin role

* Plugin management requires admin role

* Clean Opal projects web services

* Helper class to read a taxonomy in YAML format from a stream

* Other study table source handled in the admin
ymarcon added a commit to obiba/mica-search-es that referenced this issue Mar 14, 2023
@ymarcon
Properties 'project' and 'table' replaced by 'source' to support diff…
53b427c 
…erent variable origins (not necessarily opal) obiba/mica2#4376
kazoompa pushed a commit to obiba/mica-search-es that referenced this issue Jul 21, 2023
@ymarcon
Properties 'project' and 'table' replaced by 'source' to support diff…
0e68f2e 
…erent variable origins (not necessarily opal) obiba/mica2#4376
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ymarcon ymarcon

Labels
new feature
Projects
NFDI4Health
Status: Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ymarcon