db upgrage: upgrade postgres 15 to postgres 16 #1415

vh05 · 2024-08-19T11:24:20Z

The Postgres upgrade from 15 to 16 or any later versions can be done by adding the env variable named POSTGRES_UPGRADE = copy in the pg db pod.

More about upgrade process is here: https://www.postgresql.org/docs/current/pgupgrade.html

In our case, we achieve the upgrade by setting this env var in noobaa-db-pg STS.

Once after upgrade is done, we have to remove the POSTGRES_UPGRADE env from the STS. Otherwise pg db pod fails to come up because the db pod check PG directory version against upgrade image version and exits if there is no upgrade required and POSTGRES_UPGRADE is still provided.

To remove this env, we store the upgrade status in noobaa-core CR after the PG upgrade and refer the same status during db reconcile. If the upgrade is already done, we remove this env from desired sts status.

This will help to upgrade the db smoothly

dannyzaken · 2024-10-07T11:02:45Z

@vh05, can you please improve the PR description? What are the changes, technical details on the env used, etc.

vh05 · 2024-10-07T11:04:28Z

@vh05, can you please improve the PR description? What are the changes, technical details on the env used, etc.

sure

pkg/system/phase2_creating.go

deploy/internal/statefulset-postgres-db.yaml

dannyzaken

I think that instead of deciding on upgrade and applying changes inside SetDesiredNooBaaDB it will be more manageable, easy to follow, and less error-prone if you implement an UpgradeDB function that will be performed before the DB reconciliation. This method will ensure that everything is upgraded correctly, and will only return nil if the reconciliation is ready to proceed.

vh05 · 2024-11-12T06:29:22Z

I think that instead of deciding on upgrade and applying changes inside SetDesiredNooBaaDB it will be more manageable, easy to follow, and less error-prone if you implement an UpgradeDB function that will be performed before the DB reconciliation. This method will ensure that everything is upgraded correctly, and will only return nil if the reconciliation is ready to proceed.

Sure Danny. Let me check and do the changes

pkg/system/phase2_creating.go

pkg/apis/noobaa/v1alpha1/noobaa_types.go

pkg/system/system.go

dannyzaken · 2024-11-19T10:33:41Z

pkg/system/phase2_creating.go

+	// Note: Intention of the below call is to fetch the desired image from Noobaa CR or env var
+	// Passing empty Arg "" to GetDesiredDBImage() in the below case
+	desiredImage := GetDesiredDBImage(r.NooBaa, "")
+	if desiredImage != options.DBImage {


I don't understand this condition. options.DBImage is hardcoded to the upstream DB image.

We are only going ahead with upgrade if the desired image is what is provided in options.DBImage. If not we will continue with the desired image. With this condition, we only upgrade to PG16.

Do we have to consider upgrade whateven the desired image comes to be ?
ex: If desired image comes to be as postgres15, Do we have to consider postgres15 for upgrade by checking against the sts image ? not the options.DBImage ?

For the downstream the desired image will definitely be different than quay.io/sclorg/postgresql-16-c9s

Yes, on downstream, it will be registry.redhat.io/rhel9/postgresql-16

For downstream build, dont we use above image for options.DBImage ?

Apologies. I thought we will go for whatever is there in options.DBImage for upgrade. Got clarity after discussing with you that we have to match the noobaa db image and use the image given in the noobaa CR. Updating the PR according to that

dannyzaken · 2024-11-20T06:27:45Z

pkg/system/phase2_creating.go

+	// Note: Intention of the below call is to fetch the desired image from Noobaa CR or env var
+	// Passing empty Arg "" to GetDesiredDBImage() in the below case
+	desiredImage := GetDesiredDBImage(r.NooBaa, "")
+	if !strings.Contains(desiredImage, "postgresql-16") {


what is the name of the DS image? can we rely on this comparison?

As an alternative for the image name comparison, you can first compare the desired image to the one set in the DB STS:

if the images are the same, nothing to do

if they differ AND the current major version is not 16 yet, start the upgrade and modify the upgrade phase

Both upstream and downstream image names contain the string postgresql-16"
upstream: quay.io/sclorg/postgresql-16-c9s
downstream:registry.redhat.io/rhel9/postgresql-16

We are doing the alternative image comparison in Switch case: "", function named: isDBUpToDate().

Even if we remove this, upgrade will be checked in case: ""

I will rename it to something like isPostgressUpgradeReqd() and use it as appliable

isPostgresUpgradeReqd() does

if the images are the same, nothing to do if they differ AND the current major version is not 16 yet, start the upgrade and modify the upgrade phase

This check is done in 2 cases

case "": This is for the 1st time when we upgrade to the new build, the PostgresUpdatePhase is empty

case "UpgradePhaseNone": This check is done everytime until the Postgres is upgrade to 16. As soon as desired image is postgres-16, we upgrade to postgres-16 and set the status to upgrade finished.

dannyzaken · 2024-11-20T06:28:29Z

pkg/system/phase2_creating.go

+	// Check the desired image, if it is not as same as db image to upgrade,
+	// don't consider for upgrade
+	// Note: Intention of the below call is to fetch the desired image from Noobaa CR or env var
+	// Passing empty Arg "" to GetDesiredDBImage() in the below case


As a first condition, check the postgres major version

The check is added in isPostgresUpgradeReqd()

The Postgres upgrade from 15 to 16 or any later versions can be done by adding the env variable named POSTGRES_UPGRADE = copy in the pg db pod. More about upgrade process is here: https://www.postgresql.org/docs/current/pgupgrade.html In our case, we achieve the upgrade by setting this env var in noobaa-db-pg STS. Once after upgrade is done, we have to remove the POSTGRES_UPGRADE env from the STS. Otherwise pg db pod fails to come up because the db pod check PG directory version against upgrade image version and exits if there is no upgrade required and POSTGRES_UPGRADE is still provided. To remove this env, we store the upgrade status in noobaa-core CR after the PG upgrade and refer the same status during db reconcile. If the upgrade is already done, we remove this env from desired sts status. This will help to upgrade the db smoothly. Signed-off-by: Vinayak Hariharmath <[email protected]>

vh05 requested a review from dannyzaken August 19, 2024 11:24

pull-request-size bot added the size/XS label Aug 19, 2024

dannyzaken marked this pull request as draft August 20, 2024 16:22

vh05 force-pushed the pg16 branch from 90011a3 to 124e9ce Compare August 22, 2024 12:13

vh05 force-pushed the pg16 branch from 124e9ce to e75d121 Compare September 30, 2024 06:10

pull-request-size bot added size/M and removed size/XS labels Sep 30, 2024

vh05 force-pushed the pg16 branch from e75d121 to 97f367b Compare September 30, 2024 06:12

vh05 force-pushed the pg16 branch from 97f367b to dfbcf15 Compare October 7, 2024 10:04

vh05 marked this pull request as ready for review October 7, 2024 10:05

vh05 requested review from jackyalbo and liranmauda October 7, 2024 10:06

vh05 force-pushed the pg16 branch 2 times, most recently from ec79248 to 009642f Compare October 22, 2024 08:26

pull-request-size bot added size/L and removed size/M labels Oct 22, 2024

vh05 force-pushed the pg16 branch from 009642f to 1db13e5 Compare November 5, 2024 05:13

dannyzaken reviewed Nov 11, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 11, 2024

View reviewed changes

deploy/internal/statefulset-postgres-db.yaml Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 11, 2024

View reviewed changes

vh05 force-pushed the pg16 branch 6 times, most recently from 999ccc6 to e0c629d Compare November 18, 2024 04:42

vh05 requested a review from dannyzaken November 18, 2024 05:58

dannyzaken reviewed Nov 18, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 18, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 18, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 18, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

dannyzaken reviewed Nov 18, 2024

View reviewed changes

pkg/system/phase2_creating.go Outdated Show resolved Hide resolved

vh05 force-pushed the pg16 branch 4 times, most recently from a7aa91c to 2155c02 Compare November 19, 2024 06:03

vh05 requested a review from dannyzaken November 19, 2024 06:13

dannyzaken reviewed Nov 19, 2024

View reviewed changes

pkg/apis/noobaa/v1alpha1/noobaa_types.go Outdated Show resolved Hide resolved

vh05 force-pushed the pg16 branch from 2155c02 to 468960a Compare November 19, 2024 08:30

vh05 requested a review from dannyzaken November 19, 2024 08:31

dannyzaken reviewed Nov 19, 2024

View reviewed changes

pkg/system/system.go Outdated Show resolved Hide resolved

vh05 force-pushed the pg16 branch from 468960a to 3d3ec28 Compare November 19, 2024 10:07

dannyzaken reviewed Nov 19, 2024

View reviewed changes

vh05 force-pushed the pg16 branch 3 times, most recently from 5307eec to 7873068 Compare November 20, 2024 03:01

vh05 requested a review from dannyzaken November 20, 2024 03:15

dannyzaken reviewed Nov 20, 2024

View reviewed changes

vh05 force-pushed the pg16 branch 5 times, most recently from 270cfd9 to 88d449c Compare November 21, 2024 06:17

vh05 force-pushed the pg16 branch from 88d449c to 4526d87 Compare November 22, 2024 09:31

vh05 requested a review from dannyzaken November 22, 2024 09:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

db upgrage: upgrade postgres 15 to postgres 16 #1415

db upgrage: upgrade postgres 15 to postgres 16 #1415

vh05 commented Aug 19, 2024 •

edited

Loading

dannyzaken commented Oct 7, 2024

vh05 commented Oct 7, 2024

dannyzaken left a comment

vh05 commented Nov 12, 2024

dannyzaken Nov 19, 2024

vh05 Nov 19, 2024 •

edited

Loading

dannyzaken Nov 19, 2024

vh05 Nov 19, 2024 •

edited

Loading

vh05 Nov 20, 2024 •

edited

Loading

dannyzaken Nov 20, 2024

vh05 Nov 20, 2024 •

edited

Loading

vh05 Nov 22, 2024

dannyzaken Nov 20, 2024

vh05 Nov 20, 2024

vh05 Nov 22, 2024

db upgrage: upgrade postgres 15 to postgres 16 #1415

Are you sure you want to change the base?

db upgrage: upgrade postgres 15 to postgres 16 #1415

Conversation

vh05 commented Aug 19, 2024 • edited Loading

dannyzaken commented Oct 7, 2024

vh05 commented Oct 7, 2024

dannyzaken left a comment

Choose a reason for hiding this comment

vh05 commented Nov 12, 2024

dannyzaken Nov 19, 2024

Choose a reason for hiding this comment

vh05 Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

dannyzaken Nov 19, 2024

Choose a reason for hiding this comment

vh05 Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

vh05 Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

dannyzaken Nov 20, 2024

Choose a reason for hiding this comment

vh05 Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

vh05 Nov 22, 2024

Choose a reason for hiding this comment

dannyzaken Nov 20, 2024

Choose a reason for hiding this comment

vh05 Nov 20, 2024

Choose a reason for hiding this comment

vh05 Nov 22, 2024

Choose a reason for hiding this comment

vh05 commented Aug 19, 2024 •

edited

Loading

vh05 Nov 19, 2024 •

edited

Loading

vh05 Nov 19, 2024 •

edited

Loading

vh05 Nov 20, 2024 •

edited

Loading

vh05 Nov 20, 2024 •

edited

Loading