Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add instructions for deployment using GitLab pipelines #1047

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
2ec5f3c
adds gitlab yml file
renato-rudnicki Aug 31, 2023
d5b4b83
adds files for gitlab
renato-rudnicki Sep 1, 2023
e041dc6
adds double quote for variable
renato-rudnicki Sep 1, 2023
67e07d0
fix lint issues
daniel-cit Sep 5, 2023
19347cd
add support to tf-wrapper
renato-rudnicki Sep 5, 2023
1b29c63
add plan job
renato-rudnicki Sep 8, 2023
fb55169
add oidc module and GitLab infrastructure
daniel-cit Sep 15, 2023
6418984
remove extra claims
daniel-cit Sep 15, 2023
c544707
update yaml file for gitlab
renato-rudnicki Sep 19, 2023
86e69f3
update yml file
renato-rudnicki Sep 21, 2023
393f30a
WIP - adds steps to create runner instance for gitlab
renato-rudnicki Sep 21, 2023
b696ecf
changes image path
renato-rudnicki Sep 29, 2023
dc438e8
wip - changes for runner config using mig
renato-rudnicki Sep 29, 2023
e608d64
Adds gitlab runner submodule
renato-rudnicki Oct 11, 2023
db95074
Adds Runner submodule
renato-rudnicki Oct 11, 2023
078e8be
fixes for gitlab-runner module
renato-rudnicki Oct 18, 2023
2f8c161
add instructions for gitlab runner
renato-rudnicki Oct 23, 2023
2340ff7
updating code for gitlab
renato-rudnicki Oct 26, 2023
2113d24
fix path for gitlab-ci.yml file
renato-rudnicki Oct 26, 2023
c228eab
rename gitlab file
renato-rudnicki Oct 26, 2023
f133dbc
update network values
renato-rudnicki Oct 27, 2023
60d3694
update readme instructions
renato-rudnicki Oct 27, 2023
0af2196
update default value for gitlab subnetwork
renato-rudnicki Oct 27, 2023
c147267
update image path
renato-rudnicki Oct 27, 2023
05b3a42
update readme
renato-rudnicki Oct 30, 2023
9b86832
review fixes
daniel-cit Oct 31, 2023
74e6b1f
Update README-GitLab.md
renato-rudnicki Oct 31, 2023
7cf9595
Delete build/gitlab/main.tf
renato-rudnicki Oct 31, 2023
92481a8
Delete build/gitlab/Dockerfile
renato-rudnicki Oct 31, 2023
58ec729
Create run _gcp_auth.sh
renato-rudnicki Oct 31, 2023
6d57937
Delete build/gitlab directory
renato-rudnicki Oct 31, 2023
b4029c2
Rename run _gcp_auth.sh to run_gcp_auth.sh
renato-rudnicki Oct 31, 2023
e3e816b
Update startup_script.sh
renato-rudnicki Oct 31, 2023
a5c956f
Delete 0-bootstrap/scripts/gl_runner.sh
renato-rudnicki Oct 31, 2023
a9334f4
Update variables.tf
renato-rudnicki Oct 31, 2023
6643abd
Update README-GitLab.md
renato-rudnicki Oct 31, 2023
9522214
fix steps
renato-rudnicki Nov 1, 2023
cce1761
update instructions for readme and troubleshooting
renato-rudnicki Nov 3, 2023
2d7a15d
service private connect and small changes
renato-rudnicki Nov 10, 2023
d5ca962
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Nov 22, 2023
2be1e3f
feat: deploy foundation with gitlab
daniel-cit Nov 24, 2023
e34181d
fixes after step by step deploy up to 4-projects
daniel-cit Nov 30, 2023
742f84f
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Nov 30, 2023
9d4fa6f
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Dec 4, 2023
0425f88
fix merge conflict
daniel-cit Dec 4, 2023
46e4c93
create branches before deploy and set branches as protected
daniel-cit Dec 5, 2023
aea1281
set variable protection to false to run plan/validate in merge requests
daniel-cit Dec 6, 2023
c0d9f11
update instructions
daniel-cit Dec 6, 2023
d52acbe
fix link to gitlab issues
daniel-cit Dec 6, 2023
72b1dd5
prevent duplicated build pipelines
daniel-cit Dec 6, 2023
5b6df62
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Dec 6, 2023
70771a1
add flag for gitlab target
daniel-cit Dec 6, 2023
6db3c85
restore default deploy method
daniel-cit Dec 6, 2023
9a00cfb
fix typo
daniel-cit Dec 6, 2023
ed375af
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Dec 6, 2023
aaadef8
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 11, 2023
aae002c
Merge branch 'master' into deploy-foundation-using-gitlab
apeabody Dec 15, 2023
4148fec
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 16, 2023
be0e862
Merge remote-tracking branch 'origin/master' into deploy-foundation-u…
daniel-cit Dec 18, 2023
4492e9d
fix REPLACE_ME parameter in validate-requirements.sh
renato-rudnicki Dec 19, 2023
9dbd976
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 20, 2023
09c755d
update bootstrap README with instructions for the usage of alternativ…
daniel-cit Dec 20, 2023
9eb9d5c
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 20, 2023
7511dda
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 20, 2023
5046495
Retry policy for TagValue error
Samir-Cit Dec 21, 2023
1e1d540
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 23, 2023
47cd5a5
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 27, 2023
02d348d
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 27, 2023
70be3a1
Merge branch 'master' into deploy-foundation-using-gitlab
daniel-cit Dec 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
895 changes: 895 additions & 0 deletions 0-bootstrap/README-GitLab.md

Large diffs are not rendered by default.

17 changes: 15 additions & 2 deletions 0-bootstrap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,7 @@ See [troubleshooting](../docs/TROUBLESHOOTING.md) if you run into issues during

## Deploying with Jenkins

If you are using the `jenkins_bootstrap` sub-module, see
[README-Jenkins](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README-Jenkins.md)
If you are using the `jenkins_bootstrap` sub-module, see [README-Jenkins](./README-Jenkins.md)
for requirements and instructions on how to run the 0-bootstrap step. Using
Jenkins requires a few manual steps, including configuring connectivity with
your current Jenkins manager (controller) environment.
Expand All @@ -129,8 +128,22 @@ If you are deploying using [GitHub Actions](https://docs.github.com/en/actions),
for requirements and instructions on how to run the 0-bootstrap step.
Using GitHub Actions requires manual creation of the GitHub repositories used in each stage.

## Deploying with GitLab Pipelines

If you are deploying using [GitLab Pipelines](https://docs.gitlab.com/ee/ci/pipelines/), see [README-GitLab.md](./README-GitLab.md)
for requirements and instructions on how to run the 0-bootstrap step.
Using GitLab Pipeline requires manual creation of the GitLab projects (repositories) used in each stage.

## Deploying with Terraform Cloud

If you are deploying using [Terraform Cloud](https://developer.hashicorp.com/terraform/cloud-docs), see [README-Terraform-Cloud.md](./README-Terraform-Cloud.md)
for requirements and instructions on how to run the 0-bootstrap step.
Using Terraform Cloud requires manual creation of the GitHub repositories or GitLab projects used in each stage.

## Deploying with Cloud Build

**Note:** When deploying with cloud build is also possible to use a [script helper](../helpers/foundation-deployer/README.md) to do the deploy.

1. Clone [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation) into your local environment and navigate to the `0-bootstrap` folder.

```bash
Expand Down
175 changes: 175 additions & 0 deletions 0-bootstrap/gitlab.tf.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

locals {
cicd_project_id = module.gitlab_cicd.project_id

wif_sa_cicd_project = {
"bootstrap" = [
"roles/iam.workloadIdentityPoolAdmin",
],
}

gl_config = {
"bootstrap" = var.gl_repos.bootstrap,
"org" = var.gl_repos.organization,
"env" = var.gl_repos.environments,
"net" = var.gl_repos.networks,
"proj" = var.gl_repos.projects,
}

gl_branch_protection_envs = {
"env" = var.gl_repos.environments,
"net" = var.gl_repos.networks,
"proj" = var.gl_repos.projects,
}

sa_mapping = {
for k, v in local.gl_config : k => {
sa_name = google_service_account.terraform-env-sa[k].name
attribute = "attribute.project_path/${var.gl_repos.owner}/${v}"
}
}

common_vars = {
"PROJECT_ID" : module.gitlab_cicd.project_id,
"CICD_RUNNER_REPO" : var.gl_repos.cicd_runner,
"WIF_PROVIDER_NAME" : module.gitlab_oidc.provider_name,
"TF_BACKEND" : module.seed_bootstrap.gcs_bucket_tfstate,
"TF_VAR_gitlab_token" : var.gitlab_token,
}

vars_list = flatten([
for k, v in local.gl_config : [
for name, value in local.common_vars : {
config = k
name = name
value = value
repository = v
}
]
])

sa_vars = [for k, v in local.gl_config : {
config = k
name = "SERVICE_ACCOUNT_EMAIL"
value = google_service_account.terraform-env-sa[k].email
repository = v
}
]

gl_vars = { for v in concat(local.sa_vars, local.vars_list) : "${v.config}.${v.name}" => v }

}

provider "gitlab" {
token = var.gitlab_token
}

module "gitlab_cicd" {
source = "terraform-google-modules/project-factory/google"
version = "~> 12.0"

name = "${var.project_prefix}-b-cicd-wif-gl"
random_project_id = true
org_id = var.org_id
folder_id = google_folder.bootstrap.id
billing_account = var.billing_account
activate_apis = [
"compute.googleapis.com",
"admin.googleapis.com",
"iam.googleapis.com",
"billingbudgets.googleapis.com",
"cloudbilling.googleapis.com",
"serviceusage.googleapis.com",
"cloudresourcemanager.googleapis.com",
"iamcredentials.googleapis.com",
"sts.googleapis.com",
"dns.googleapis.com",
"secretmanager.googleapis.com",

]
}

module "gitlab_oidc" {
source = "./modules/gitlab-oidc"

project_id = module.gitlab_cicd.project_id
pool_id = "foundation-pool"
provider_id = "foundation-gl-provider"
sa_mapping = local.sa_mapping
}

resource "gitlab_project_variable" "variables" {
for_each = local.gl_vars

project = "${var.gl_repos.owner}/${each.value.repository}"
key = each.value.name
value = each.value.value
protected = false
masked = true
}

resource "gitlab_branch_protection" "image" {
project = "${var.gl_repos.owner}/${var.gl_repos.cicd_runner}"
branch = "image"
}

resource "gitlab_branch_protection" "plan" {
for_each = local.gl_config

project = "${var.gl_repos.owner}/${each.value}"
branch = "plan"
}

resource "gitlab_branch_protection" "production" {
for_each = local.gl_config

project = "${var.gl_repos.owner}/${each.value}"
branch = "production"
}

resource "gitlab_branch_protection" "non_production" {
for_each = local.gl_branch_protection_envs

project = "${var.gl_repos.owner}/${each.value}"
branch = "non-production"
}

resource "gitlab_branch_protection" "development" {
for_each = local.gl_branch_protection_envs

project = "${var.gl_repos.owner}/${each.value}"
branch = "development"
}

module "cicd_project_wif_iam_member" {
source = "./modules/parent-iam-member"
for_each = local.wif_sa_cicd_project

member = "serviceAccount:${google_service_account.terraform-env-sa[each.key].email}"
parent_type = "project"
parent_id = local.cicd_project_id
roles = each.value
}

resource "google_service_account_iam_member" "self_impersonate" {
for_each = local.granular_sa

service_account_id = google_service_account.terraform-env-sa[each.key].id
role = "roles/iam.serviceAccountTokenCreator"
member = "serviceAccount:${google_service_account.terraform-env-sa[each.key].email}"
}
51 changes: 51 additions & 0 deletions 0-bootstrap/modules/gitlab-oidc/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

resource "google_project_service" "services" {
project = var.project_id
count = length(var.service_list)
service = var.service_list[count.index]
disable_on_destroy = false
}

resource "google_iam_workload_identity_pool" "main" {
project = var.project_id
workload_identity_pool_id = var.pool_id
display_name = var.pool_display_name
description = var.pool_description
disabled = false
}

resource "google_iam_workload_identity_pool_provider" "main" {
project = var.project_id
workload_identity_pool_id = google_iam_workload_identity_pool.main.workload_identity_pool_id
workload_identity_pool_provider_id = var.provider_id
display_name = var.provider_display_name
description = var.provider_description
attribute_condition = var.attribute_condition
attribute_mapping = var.attribute_mapping
oidc {
allowed_audiences = var.allowed_audiences
issuer_uri = var.issuer_uri
}
}

resource "google_service_account_iam_member" "wif-sa" {
for_each = var.sa_mapping
service_account_id = each.value.sa_name
role = "roles/iam.workloadIdentityUser"
member = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.main.name}/${each.value.attribute}"
}
25 changes: 25 additions & 0 deletions 0-bootstrap/modules/gitlab-oidc/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/**
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

output "pool_name" {
description = "Pool name"
value = google_iam_workload_identity_pool.main.name
}

output "provider_name" {
description = "Provider name"
value = google_iam_workload_identity_pool_provider.main.name
}
Loading