Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DALA-4584 Introduce Dataland Lego Bricks #812

Open
wants to merge 97 commits into
base: main
Choose a base branch
from
Open

DALA-4584 Introduce Dataland Lego Bricks #812

wants to merge 97 commits into from

Conversation

MarcTM01
Copy link
Contributor

Pull Request <Title>

<Description here>

Things to do during Peer Review

Please check all boxes before the Pull Request is merged. In case you skip a box, describe in the PRs description (that means: here) why the check is skipped.

  • The GitHub Actions (including Sonarqube Gateway and Lint Checks) are green. This is enforced by GitHub.
  • The PR has been peer-reviewed
    • The code has been manually inspected by someone who did not implement the feature
    • If this PR includes work on the frontend, at least one @ts-nocheck is removed. Additionally, there should not be any @ts-nocheck in files modified by this PR. If no @ts-nocheck are left: Celebrate 🎉 🎊 type-safety and remove this entry.
  • The PR actually implements what is described in the JIRA-Issue
  • At least one test exists testing the new feature
    • If you have created new test files, make sure that they are included in a test container and actually run in the CI
  • At least 3 consecutive CI runs are successfully executed to ensure that there are no flaky tests.
  • Documentation is updated as required
  • The automated deployment is updated if required
  • If there was a database entity class added, there must also be a migration script for creating the corresponding database if flyway is already used by the service
  • IF there are changes done to the framework data models or to a database entity class, the following steps were completed in order
    • A fresh clone of dataland.com is generated (see Wiki page on "OTC" for details)
    • The feature branch is deployed to clone with Reset non-user related Docker Volumes & Re-populate turned off
    • It's verified that the CD run is green
    • The new feature is manually used/tested/observed on the clone server. Testing of the new feature should (also) be done by a second, independent reviewer from the dev team
    • The feature branch is deployed to dev1 with Reset non-user related Docker Volumes & Re-populate turned on, and it's verified that the CD run is green
  • ELSE, the new version is deployed to the dev server "dev1" using this branch
    • Run with setting Reset non-user related Docker Volumes & Re-populate turned on
    • It's verified that this version actually is the one deployed (check gitinfo for branch name and commit id!)
    • It's verified that the CD run is green
    • The new feature is manually used/tested/observed on dev server. Testing of the new feature should (also) be done by a second, independent reviewer from the dev team
  • Confirm that the latest version of the feature is deployed to the dev server "dev1" using this branch (no longer enforced by GitHub)

dennis-dfine and others added 30 commits October 24, 2024 18:04
@dennis-dfine
Create Specification Service
356b728
@SiegfriedSobkowiak
add first set of files
2738e00
@SiegfriedSobkowiak
fix error
a0b0ce4
@dennis-dfine
Add entrypoint script
8f67c03
@dennis-dfine
Add datapoints and datatypes for additional-company-information
c61600b
@SiegfriedSobkowiak
Add datapoint validation
2aba6da
@SiegfriedSobkowiak
Merge remote-tracking branch 'origin/main' into DALA-4584_Introduce-L…
66b8e14 
…egos
@SiegfriedSobkowiak
Add up- and download for data points into in memory store
8cac4bd
@dennis-dfine
Merge remote-tracking branch 'origin/main' into DALA-4584_Introduce-L…
1911747 
…egos
@dennis-dfine
Update SpecificationService
00654af
@dennis-dfine
Add Todos Naming
2e08fe9
@SiegfriedSobkowiak
Store data-point in the same structure as the data sets before
7ad96ae
@SiegfriedSobkowiak
Merge remote-tracking branch 'origin/DALA-4584_Introduce-Legos' into …
be7b68d 
…DALA-4584_Introduce-Legos

# Conflicts:
#	dataland-backend/src/main/kotlin/org/dataland/datalandbackend/services/DataPointManager.kt
@SiegfriedSobkowiak
add todo
e29a196
@SiegfriedSobkowiak
Merge remote-tracking branch 'origin/main' into DALA-4584_Introduce-L…
9ebbe58 
…egos

# Conflicts:
#	dataland-backend/backendOpenApi.json
@SiegfriedSobkowiak
add open api spec
daea79a
@dennis-dfine
Fix links in specifications
0126038
@dennis-dfine
Fix automated QA Service
3d43367
@dennis-dfine
Update backend to match specifications
ed47dc8
@dennis-dfine
Merge branch 'DALA-4584_Introduce-Legos' of github.com:d-fine/Datalan…
d9cec7d 
…d into DALA-4584_Introduce-Legos
@SiegfriedSobkowiak
update naming
68e9ec0
@dennis-dfine
Fix error in automated QA service
83e28f9
@dennis-dfine
Modify internal storage to store datapoints
c2b753c
@MarcTM01
Transform Specification Service into fully-featured spring service
e057a87
@MarcTM01
Transform Specification Service into fully-featured spring service (P…
1a68812 
…art 2)
@dennis-dfine
Finally fix automated QA Service
944b3d0
@SiegfriedSobkowiak
introduce breaking down
2a21031
@MarcTM01
Introduce logic to the specification service.
626d37d
@MarcTM01
Merge remote-tracking branch 'origin/DALA-4584_Introduce-Legos' into …
2a8b3b8 
…DALA-4584_Introduce-Legos
@MarcTM01
Introduce missing endpoints to specification service. Generated OpenA…
1fc1ca8 
…pi Spec.
MarcTM01
MarcTM01 commented Nov 12, 2024
View reviewed changes
dataland-automated-qa-service/src/main/infrastructure/resources/data_resource.py Show resolved Hide resolved
dataland-automated-qa-service/src/main/infrastructure/resources/data_resource.py Show resolved Hide resolved
dataland-backend/src/main/kotlin/org/dataland/datalandbackend/controller/DataController.kt Outdated Show resolved Hide resolved
dataland-backend/src/main/kotlin/org/dataland/datalandbackend/controller/DataController.kt Outdated Show resolved Hide resolved
dataland-backend/src/main/kotlin/org/dataland/datalandbackend/model/DataType.kt
@@ -46,14 +45,5 @@ data class DataType
get() = allowedDataTypes.map { valueOf(it) }
}

init {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this may have unwanted side-effects.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo: revisit after separation of frameworks and data points is implemented

dataland-backend/src/main/kotlin/org/dataland/datalandbackend/validator/ValueCurrency.kt Outdated Show resolved Hide resolved
dataland-backend/src/test/resources/json/frameworkTemplate/referencedReports.json Outdated Show resolved Hide resolved
dataland-frontend/tests/e2e/utils/GeneralUtils.ts Outdated Show resolved Hide resolved
dataland-frontend/tests/e2e/utils/GeneralUtils.ts Outdated Show resolved Hide resolved
...age/src/main/kotlin/org/dataland/datalandinternalstorage/services/DatabaseStringDataStore.kt
logger.info("Inserting data into database with data ID: $dataId and correlation ID: $correlationId.")
storeDataItemWithoutTransaction(DataItem(dataId, objectMapper.writeValueAsString(data)))

val dataAsJSON = JSONObject(dataObject)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entirely breaks our current assumption of the Internal Storage: So far, internal storage always was a "dumb" Key-Value Store. IMO we should carefully consider implement such logic here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SiegfriedSobkowiak SiegfriedSobkowiak SiegfriedSobkowiak left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MarcTM01 @SiegfriedSobkowiak @dennis-dfine