feat: add support for fine grained configuration of VPC-flow logs

0-bootstrap/modules/cb-private-pool/README.md

@@ -5,6 +5,7 @@
 |------|-------------|------|---------|:--------:|
 | private\_worker\_pool | name: Name of the worker pool. A name with a random suffix is generated if not set.<br>  region: The private worker pool region. See https://cloud.google.com/build/docs/locations for available locations.<br>  disk\_size\_gb: Size of the disk attached to the worker, in GB.<br>  machine\_type: Machine type of a worker.<br>  no\_external\_ip: If true, workers are created without any public address, which prevents network egress to public IPs.<br>  enable\_network\_peering: Set to true to enable configuration of networking peering for the private worker pool.<br>  create\_peered\_network: If true a network will be created to stablish the network peering.<br>  peered\_network\_id: The ID of the existing network to configure peering for the private worker pool if create\_peered\_network false. The project containing the network must have Service Networking API (`servicenetworking.googleapis.com`) enabled.<br>  peered\_network\_subnet\_ip: The IP range to be used for the subnet that a will created in the peered network if create\_peered\_network true.<br>  peering\_address: The IP address or beginning of the peering address range. This can be supplied as an input to reserve a specific address or omitted to allow GCP to choose a valid one.<br>  peering\_prefix\_length: The prefix length of the IP peering range. If not present, it means the address field is a single IP address. | <pre>object({<br>    name                     = optional(string, "")<br>    region                   = optional(string, "us-central1")<br>    disk_size_gb             = optional(number, 100)<br>    machine_type             = optional(string, "e2-medium")<br>    no_external_ip           = optional(bool, false)<br>    enable_network_peering   = optional(bool, false)<br>    create_peered_network    = optional(bool, false)<br>    peered_network_id        = optional(string, "")<br>    peered_network_subnet_ip = optional(string, "")<br>    peering_address          = optional(string, null)<br>    peering_prefix_length    = optional(number, 24)<br>  })</pre> | `{}` | no |
 | project\_id | ID of the project where the private pool will be created | `string` | n/a | yes |
+| vpc\_flow\_logs | aggregation\_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL\_5\_SEC, INTERVAL\_30\_SEC, INTERVAL\_1\_MIN, INTERVAL\_5\_MIN, INTERVAL\_10\_MIN, INTERVAL\_15\_MIN.<br>  flow\_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].<br>  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE\_ALL\_METADATA, INCLUDE\_ALL\_METADATA, CUSTOM\_METADATA.<br>  metadata\_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM\_METADATA.<br>  filter\_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field. | <pre>object({<br>    aggregation_interval = optional(string, "INTERVAL_5_SEC")<br>    flow_sampling        = optional(string, "0.5")<br>    metadata             = optional(string, "INCLUDE_ALL_METADATA")<br>    metadata_fields      = optional(list(string), [])<br>    filter_expr          = optional(string, "true")<br>  })</pre> | `{}` | no |
 | vpn\_configuration | enable\_vpn: set to true to create VPN connection to on prem. If true, the following values must be valid.<br>  on\_prem\_public\_ip\_address0: The first public IP address for on prem VPN connection.<br>  on\_prem\_public\_ip\_address1: The second public IP address for on prem VPN connection.<br>  router\_asn: Border Gateway Protocol (BGP) Autonomous System Number (ASN) for cloud routes.<br>  bgp\_peer\_asn: Border Gateway Protocol (BGP) Autonomous System Number (ASN) for peer cloud routes.<br>  shared\_secret: The shared secret used in the VPN.<br>  psk\_secret\_project\_id: The ID of the project that contains the secret from secret manager that holds the VPN pre-shared key.<br>  psk\_secret\_name: The name of the secret to retrieve from secret manager that holds the VPN pre-shared key.<br>  tunnel0\_bgp\_peer\_address: BGP peer address for tunnel 0.<br>  tunnel0\_bgp\_session\_range: BGP session range for tunnel 0.<br>  tunnel1\_bgp\_peer\_address: BGP peer address for tunnel 1.<br>  tunnel1\_bgp\_session\_range: BGP session range for tunnel 1. | <pre>object({<br>    enable_vpn                 = optional(bool, false)<br>    on_prem_public_ip_address0 = optional(string, "")<br>    on_prem_public_ip_address1 = optional(string, "")<br>    router_asn                 = optional(number, 64515)<br>    bgp_peer_asn               = optional(number, 64513)<br>    psk_secret_project_id      = optional(string, "")<br>    psk_secret_name            = optional(string, "")<br>    tunnel0_bgp_peer_address   = optional(string, "")<br>    tunnel0_bgp_session_range  = optional(string, "")<br>    tunnel1_bgp_peer_address   = optional(string, "")<br>    tunnel1_bgp_session_range  = optional(string, "")<br>  })</pre> | `{}` | no |
 
 ## Outputs

0-bootstrap/modules/cb-private-pool/network.tf

@@ -20,7 +20,7 @@ locals {
 
 module "peered_network" {
   source  = "terraform-google-modules/network/google"
-  version = "~> 7.0"
+  version = "~> 8.0"
   count   = var.private_worker_pool.create_peered_network ? 1 : 0
 
   project_id                             = var.project_id
@@ -29,12 +29,17 @@ module "peered_network" {
 
   subnets = [
     {
-      subnet_name           = "sb-b-cbpools-${var.private_worker_pool.region}"
-      subnet_ip             = var.private_worker_pool.peered_network_subnet_ip
-      subnet_region         = var.private_worker_pool.region
-      subnet_private_access = "true"
-      subnet_flow_logs      = "true"
-      description           = "Peered subnet for Cloud Build private pool"
+      subnet_name                      = "sb-b-cbpools-${var.private_worker_pool.region}"
+      subnet_ip                        = var.private_worker_pool.peered_network_subnet_ip
+      subnet_region                    = var.private_worker_pool.region
+      subnet_private_access            = "true"
+      subnet_flow_logs                 = "true"
+      subnet_flow_logs_interval        = var.vpc_flow_logs.aggregation_interval
+      subnet_flow_logs_sampling        = var.vpc_flow_logs.flow_sampling
+      subnet_flow_logs_metadata        = var.vpc_flow_logs.metadata
+      subnet_flow_logs_metadata_fields = var.vpc_flow_logs.metadata_fields
+      subnet_flow_logs_filter          = var.vpc_flow_logs.filter_expr
+      description                      = "Peered subnet for Cloud Build private pool"
     }
   ]
 

0-bootstrap/modules/cb-private-pool/variables.tf

@@ -106,3 +106,21 @@ variable "vpn_configuration" {
     error_message = "If VPN configuration is enabled, all values are required."
   }
 }
+
+variable "vpc_flow_logs" {
+  description = <<EOT
+  aggregation_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL_5_SEC, INTERVAL_30_SEC, INTERVAL_1_MIN, INTERVAL_5_MIN, INTERVAL_10_MIN, INTERVAL_15_MIN.
+  flow_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].
+  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE_ALL_METADATA, INCLUDE_ALL_METADATA, CUSTOM_METADATA.
+  metadata_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM_METADATA.
+  filter_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field.
+EOT
+  type = object({
+    aggregation_interval = optional(string, "INTERVAL_5_SEC")
+    flow_sampling        = optional(string, "0.5")
+    metadata             = optional(string, "INCLUDE_ALL_METADATA")
+    metadata_fields      = optional(list(string), [])
+    filter_expr          = optional(string, "true")
+  })
+  default = {}
+}

0-bootstrap/modules/jenkins-agent/main.tf

@@ -143,6 +143,7 @@ resource "google_compute_subnetwork" "jenkins_agents_subnet" {
     aggregation_interval = "INTERVAL_5_SEC"
     flow_sampling        = 0.5
     metadata             = "INCLUDE_ALL_METADATA"
+    metadata_fields      = null
     filter_expr          = true
   }
 }

3-networks-dual-svpc/envs/shared/README.md

@@ -19,8 +19,8 @@ The purpose of this step is to set up the global [DNS Hub](https://cloud.google.
 | firewall\_policies\_enable\_logging | Toggle hierarchical firewall logging. | `bool` | `true` | no |
 | preactivate\_partner\_interconnect | Preactivate Partner Interconnect VLAN attachment in the environment. | `bool` | `false` | no |
 | remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes |
-| subnetworks\_enable\_logging | Toggle subnetworks flow logging for VPC Subnetworks. | `bool` | `true` | no |
 | target\_name\_server\_addresses | List of IPv4 address of target name servers for the forwarding zone configuration. See https://cloud.google.com/dns/docs/overview#dns-forwarding-zones for details on target name servers in the context of Cloud DNS forwarding zones. | `list(map(any))` | n/a | yes |
+| vpc\_flow\_logs | enable\_logging: set to true to enable VPC flow logging for the subnetworks.<br>  aggregation\_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL\_5\_SEC, INTERVAL\_30\_SEC, INTERVAL\_1\_MIN, INTERVAL\_5\_MIN, INTERVAL\_10\_MIN, INTERVAL\_15\_MIN.<br>  flow\_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].<br>  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE\_ALL\_METADATA, INCLUDE\_ALL\_METADATA, CUSTOM\_METADATA.<br>  metadata\_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM\_METADATA.<br>  filter\_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field. | <pre>object({<br>    enable_logging       = optional(string, "true")<br>    aggregation_interval = optional(string, "INTERVAL_5_SEC")<br>    flow_sampling        = optional(string, "0.5")<br>    metadata             = optional(string, "INCLUDE_ALL_METADATA")<br>    metadata_fields      = optional(list(string), [])<br>    filter_expr          = optional(string, "true")<br>  })</pre> | `{}` | no |
 
 ## Outputs
 

3-networks-dual-svpc/envs/shared/dns-hub.tf

@@ -20,27 +20,37 @@
 
 module "dns_hub_vpc" {
   source  = "terraform-google-modules/network/google"
-  version = "~> 7.0"
+  version = "~> 8.0"
 
   project_id                             = local.dns_hub_project_id
   network_name                           = "vpc-c-dns-hub"
   shared_vpc_host                        = "false"
   delete_default_internet_gateway_routes = "true"
 
   subnets = [{
-    subnet_name           = "sb-c-dns-hub-${local.default_region1}"
-    subnet_ip             = "172.16.0.0/25"
-    subnet_region         = local.default_region1
-    subnet_private_access = "true"
-    subnet_flow_logs      = var.subnetworks_enable_logging
-    description           = "DNS hub subnet for region 1."
+    subnet_name                      = "sb-c-dns-hub-${local.default_region1}"
+    subnet_ip                        = "172.16.0.0/25"
+    subnet_region                    = local.default_region1
+    subnet_private_access            = "true"
+    subnet_flow_logs                 = var.vpc_flow_logs.enable_logging
+    subnet_flow_logs_interval        = var.vpc_flow_logs.aggregation_interval
+    subnet_flow_logs_sampling        = var.vpc_flow_logs.flow_sampling
+    subnet_flow_logs_metadata        = var.vpc_flow_logs.metadata
+    subnet_flow_logs_metadata_fields = var.vpc_flow_logs.metadata_fields
+    subnet_flow_logs_filter          = var.vpc_flow_logs.filter_expr
+    description                      = "DNS hub subnet for region 1."
     }, {
-    subnet_name           = "sb-c-dns-hub-${local.default_region2}"
-    subnet_ip             = "172.16.0.128/25"
-    subnet_region         = local.default_region2
-    subnet_private_access = "true"
-    subnet_flow_logs      = var.subnetworks_enable_logging
-    description           = "DNS hub subnet for region 2."
+    subnet_name                      = "sb-c-dns-hub-${local.default_region2}"
+    subnet_ip                        = "172.16.0.128/25"
+    subnet_region                    = local.default_region2
+    subnet_private_access            = "true"
+    subnet_flow_logs                 = var.vpc_flow_logs.enable_logging
+    subnet_flow_logs_interval        = var.vpc_flow_logs.aggregation_interval
+    subnet_flow_logs_sampling        = var.vpc_flow_logs.flow_sampling
+    subnet_flow_logs_metadata        = var.vpc_flow_logs.metadata
+    subnet_flow_logs_metadata_fields = var.vpc_flow_logs.metadata_fields
+    subnet_flow_logs_filter          = var.vpc_flow_logs.filter_expr
+    description                      = "DNS hub subnet for region 2."
   }]
 
   routes = [{

3-networks-dual-svpc/envs/shared/variables.tf

@@ -25,10 +25,24 @@ variable "dns_enable_logging" {
   default     = true
 }
 
-variable "subnetworks_enable_logging" {
-  type        = bool
-  description = "Toggle subnetworks flow logging for VPC Subnetworks."
-  default     = true
+variable "vpc_flow_logs" {
+  description = <<EOT
+  enable_logging: set to true to enable VPC flow logging for the subnetworks.
+  aggregation_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL_5_SEC, INTERVAL_30_SEC, INTERVAL_1_MIN, INTERVAL_5_MIN, INTERVAL_10_MIN, INTERVAL_15_MIN.
+  flow_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].
+  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE_ALL_METADATA, INCLUDE_ALL_METADATA, CUSTOM_METADATA.
+  metadata_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM_METADATA.
+  filter_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field.
+EOT
+  type = object({
+    enable_logging       = optional(string, "true")
+    aggregation_interval = optional(string, "INTERVAL_5_SEC")
+    flow_sampling        = optional(string, "0.5")
+    metadata             = optional(string, "INCLUDE_ALL_METADATA")
+    metadata_fields      = optional(list(string), [])
+    filter_expr          = optional(string, "true")
+  })
+  default = {}
 }
 
 variable "domain" {

3-networks-dual-svpc/modules/base_env/README.md

@@ -8,6 +8,7 @@
 | base\_private\_service\_connect\_ip | The base subnet internal IP to be used as the private service connect endpoint in the Base Shared VPC | `string` | n/a | yes |
 | base\_subnet\_primary\_ranges | The base subnet primary IPTs ranges to the Base Shared Vpc. | `map(string)` | n/a | yes |
 | base\_subnet\_secondary\_ranges | The base subnet secondary IPTs ranges to the Base Shared Vpc. | `map(list(map(string)))` | n/a | yes |
+| base\_vpc\_flow\_logs | aggregation\_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL\_5\_SEC, INTERVAL\_30\_SEC, INTERVAL\_1\_MIN, INTERVAL\_5\_MIN, INTERVAL\_10\_MIN, INTERVAL\_15\_MIN.<br>  flow\_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].<br>  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE\_ALL\_METADATA, INCLUDE\_ALL\_METADATA, CUSTOM\_METADATA.<br>  metadata\_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM\_METADATA.<br>  filter\_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field. | <pre>object({<br>    aggregation_interval = optional(string, "INTERVAL_5_SEC")<br>    flow_sampling        = optional(string, "0.5")<br>    metadata             = optional(string, "INCLUDE_ALL_METADATA")<br>    metadata_fields      = optional(list(string), [])<br>    filter_expr          = optional(string, "true")<br>  })</pre> | `{}` | no |
 | custom\_restricted\_services | List of custom services to be protected by the VPC-SC perimeter. If empty, all supported services (https://cloud.google.com/vpc-service-controls/docs/supported-products) will be protected. | `list(string)` | `[]` | no |
 | default\_region1 | First subnet region. The shared vpc modules only configures two regions. | `string` | n/a | yes |
 | default\_region2 | Second subnet region. The shared vpc modules only configures two regions. | `string` | n/a | yes |
@@ -24,6 +25,7 @@
 | restricted\_private\_service\_connect\_ip | The base subnet internal IP to be used as the private service connect endpoint in the Restricted Shared VPC | `string` | n/a | yes |
 | restricted\_subnet\_primary\_ranges | The base subnet primary IPTs ranges to the Restricted Shared Vpc. | `map(string)` | n/a | yes |
 | restricted\_subnet\_secondary\_ranges | The base subnet secondary IPTs ranges to the Restricted Shared Vpc | `map(list(map(string)))` | n/a | yes |
+| restricted\_vpc\_flow\_logs | aggregation\_interval: Toggles the aggregation interval for collecting flow logs. Increasing the interval time will reduce the amount of generated flow logs for long lasting connections. Possible values are: INTERVAL\_5\_SEC, INTERVAL\_30\_SEC, INTERVAL\_1\_MIN, INTERVAL\_5\_MIN, INTERVAL\_10\_MIN, INTERVAL\_15\_MIN.<br>  flow\_sampling: Set the sampling rate of VPC flow logs within the subnetwork where 1.0 means all collected logs are reported and 0.0 means no logs are reported. The value of the field must be in [0, 1].<br>  metadata: Configures whether metadata fields should be added to the reported VPC flow logs. Possible values are: EXCLUDE\_ALL\_METADATA, INCLUDE\_ALL\_METADATA, CUSTOM\_METADATA.<br>  metadata\_fields: ist of metadata fields that should be added to reported logs. Can only be specified if VPC flow logs for this subnetwork is enabled and "metadata" is set to CUSTOM\_METADATA.<br>  filter\_expr: Export filter used to define which VPC flow logs should be logged, as as CEL expression. See https://cloud.google.com/vpc/docs/flow-logs#filtering for details on how to format this field. | <pre>object({<br>    aggregation_interval = optional(string, "INTERVAL_5_SEC")<br>    flow_sampling        = optional(string, "0.5")<br>    metadata             = optional(string, "INCLUDE_ALL_METADATA")<br>    metadata_fields      = optional(list(string), [])<br>    filter_expr          = optional(string, "true")<br>  })</pre> | `{}` | no |
 
 ## Outputs