Skip to content

Commit

Permalink
feat(node affinity): add support to specify node affinity rules of NF…
Browse files Browse the repository at this point in the history 
…S Server (#59)

* feat(node affinity): add support to specify node affinity rules of NFS Server

This commit adds support to specify node affinity rules via NFS-Provisioner
ENV to schedule scheduling NFS Server on set of nodes.

**How to use?**:
- Add 'OPENEBS_IO_NFS_SERVER_NODE_AFFINITY' ENV in NFS-Provisioner deployment in following manner:
  ```sh
  - name: OPENEBS_IO_NFS_SERVER_NODE_AFFINITY
    value: "kubernetes.io/hostname:[172.17.0.1],kubernetes.io/os:[linux]"
  ```
- To schedule NFS Server instance on storage & nfs nodes
  ```sh
  - name: OPENEBS_IO_NFS_SERVER_NODE_AFFINITY
    value: "kubernetes.io/storage,kubernetes.io/nfs-node"
  ```
  Ex: Propagation to NFS Server deployment
   ```yaml
   ...
   ...
   ...
          nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/storage-node
                operator: Exists
              - key: kubernetes.io/arch
                operator: Exists
   ```

**How it is propagated to NFS Server instance**:
- During boot-up time of provisioner instance, provisioner will read
  OPENEBS_IO_NFS_SERVER_NODE_AFFINITY ENV then parse 
  affinity rules and store them under the affinity rules in form of Go structure[in-memory].
- When volume is provisioned NFS-Provisioner will propagate this affinity
  rules to NFS Server instance.
Example propagation:
```yaml
...
...
...
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - 172.17.0.1
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
```

* add integration test case to verify node affinity rules
* helm chart changes and fixes test cases

Signed-off-by: mittachaitu <[email protected]>
  • Loading branch information
sai chaithanya authored Jul 9, 2021
1 parent af64d34 commit cbb6dc2
Show file tree
Hide file tree
Showing 16 changed files with 874 additions and 28 deletions.
2 changes: 1 addition & 1 deletion deploy/helm/charts/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Helm chart for OpenEBS Dynamic NFS PV. For instructions to install
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.4.1
version: 0.4.2
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 0.4.0
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/charts/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ helm install openebs-nfs openebs-nfs/nfs-provisioner --namespace openebs --creat
| `nfsProvisioner.securityContext` | Security context for container | `""` |
| `nfsProvisioner.tolerations` | NFS Provisioner pod toleration values | `""` |
| `nfsProvisioner.nfsServerNamespace` | NFS server namespace | `"openebs"` |
| `nfsProvisioner.nfsServerNodeAffinity` | NFS Server node affinity rules | `""` |
| `nfsStorageClass.backendStorageClass` | StorageClass to be used to provision the backend volume. If not specified, the default StorageClass is used. | `""` |
| `nfsStorageClass.isDefaultClass` | Make 'openebs-kernel-nfs' the default StorageClass | `"false"` |
| `nfsStorageClass.reclaimPolicy` | ReclaimPolicy for NFS PVs | `"Delete"` |
Expand Down
13 changes: 12 additions & 1 deletion deploy/helm/charts/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,18 @@ spec:
- name: OPENEBS_IO_NFS_SERVER_NS
value: {{ .Values.nfsProvisioner.nfsServerNamespace }}
{{- end }}

# OPENEBS_IO_NFS_SERVER_NODE_AFFINITY defines the node affinity rules to place NFS Server
# instance. It accepts affinity rules in multiple ways:
# - If NFS Server needs to be placed on storage nodes as well as only in
# zone-1 & zone-2 then value can be:
# value: "kubernetes.io/zone:[zone-1,zone-2],kubernetes.io/storage-node".
# - If NFS Server needs to be placed only on storage nodes & nfs nodes then
# value can be:
# value: "kubernetes.io/storage-node,kubernetes.io/nfs-node"
{{- if .Values.nfsProvisioner.nfsServerNodeAffinity }}
- name: OPENEBS_IO_NFS_SERVER_NODE_AFFINITY
value: "{{ .Values.nfsProvisioner.nfsServerNodeAffinity }}"
{{- end }}
# Process name used for matching is limited to the 15 characters
# present in the pgrep output.
# So fullname can't be used here with pgrep (>15 chars).A regular expression
Expand Down
14 changes: 11 additions & 3 deletions deploy/helm/charts/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,17 @@ nfsProvisioner:
healthCheck:
initialDelaySeconds: 30
periodSeconds: 60
# namespace in which nfs server objects should be created
# By default, nfs provisioner will create these resources in nfs provisioner's namespace
# nfsServerNamespace: openebs
# namespace in which nfs server objects should be created
# By default, nfs provisioner will create these resources in nfs provisioner's namespace
# nfsServerNamespace: openebs
#
# nfsServerNodeAffinity defines the node affinity rules to place NFS Server
# instance. It accepts affinity rules in multiple ways:
# - If NFS Server needs to be placed on storage nodes as well as only in
# zone-1 & zone-2 then value can be: "kubernetes.io/zone:[zone-1,zone-2],kubernetes.io/storage-node".
# - If NFS Server needs to be placed only on storage nodes & nfs nodes then
# value can be: "kubernetes.io/storage-node,kubernetes.io/nfs-node"
# nfsServerNodeAffinity: "kubernetes.io/storage-node,kubernetes.io/nfs-node"

nfsStorageClass:
name: openebs-kernel-nfs
Expand Down
34 changes: 22 additions & 12 deletions deploy/kubectl/openebs-nfs-provisioner.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,16 +92,26 @@ spec:
imagePullPolicy: IfNotPresent
image: openebs/provisioner-nfs:ci
env:
# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s
# based on this address. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_K8S_MASTER
# value: "http://10.128.0.12:8080"
# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s
# based on this config. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_KUBE_CONFIG
# value: "/home/ubuntu/.kube/config"
# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s
# based on this address. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
# - name: OPENEBS_IO_K8S_MASTER
# value: "http://10.128.0.12:8080"
# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s
# based on this config. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
# - name: OPENEBS_IO_KUBE_CONFIG
# value: "/home/ubuntu/.kube/config"
# OPENEBS_IO_NFS_SERVER_NODE_AFFINITY defines the node affinity rules to place NFS Server
# instance. It accepts affinity rules in multiple ways:
# - If NFS Server needs to be placed on storage nodes as well as only in
# zone-1 & zone-2 then value can be:
# value: "kubernetes.io/zone:[zone-1,zone-2],kubernetes.io/storage-node".
# - If NFS Server needs to be placed only on storage nodes & nfs nodes then
# value can be:
# value: "kubernetes.io/storage-node,kubernetes.io/nfs-node"
# - name: OPENEBS_IO_NFS_SERVER_NODE_AFFINITY
# value: "kubernetes.io/storage-node,kubernetes.io/nfs-node"
- name: NODE_NAME
valueFrom:
fieldRef:
Expand Down Expand Up @@ -158,8 +168,8 @@ metadata:
cas.openebs.io/config: |
- name: NFSServerType
value: "kernel"
#- name: BackendStorageClass
# value: "openebs-hostpath"
- name: BackendStorageClass
value: "openebs-hostpath"
# LeaseTime defines the renewl period(in seconds) for client state
#- name: LeaseTime
# value: 30
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,6 @@ require (
k8s.io/apimachinery v0.17.3
k8s.io/client-go v11.0.0+incompatible
k8s.io/klog v1.0.0
k8s.io/kubernetes v1.17.3
sigs.k8s.io/sig-storage-lib-external-provisioner v4.1.0+incompatible
)
1 change: 1 addition & 0 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -860,6 +860,7 @@ k8s.io/kube-proxy v0.17.3/go.mod h1:ds8R8bUYPWtQlspC47Sff7o5aQhWDsv6jpQJATDuqaQ=
k8s.io/kube-scheduler v0.17.3/go.mod h1:36HgrrPqzK+rOLTRtDG//b89KjrAZqFI4PXOpdH351M=
k8s.io/kubectl v0.17.3/go.mod h1:NUn4IBY7f7yCMwSop2HCXlw/MVYP4HJBiUmOR3n9w28=
k8s.io/kubelet v0.17.3/go.mod h1:Nh8owUHZcUXtnDAtmGnip36Nw+X6c4rbmDQlVyIhwMQ=
k8s.io/kubernetes v1.17.3 h1:zWCppkLfHM+hoLqfbsrQ0cJnYw+4vAvedI92oQnjo/Q=
k8s.io/kubernetes v1.17.3/go.mod h1:gt28rfzaskIzJ8d82TSJmGrJ0XZD0BBy8TcQvTuCI3w=
k8s.io/legacy-cloud-providers v0.17.3/go.mod h1:ujZML5v8efVQxiXXTG+nck7SjP8KhMRjUYNIsoSkYI0=
k8s.io/metrics v0.17.3/go.mod h1:HEJGy1fhHOjHggW9rMDBJBD3YuGroH3Y1pnIRw9FFaI=
Expand Down
52 changes: 52 additions & 0 deletions pkg/kubernetes/api/core/v1/podtemplatespec/podtemplatespec.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,58 @@ func (b *Builder) WithAffinity(affinity *corev1.Affinity) *Builder {
return b
}

// WithNodeAffinityMatchExpressions sets matchexpressions under
// nodeAffinity
// NOTE: If nil is passed then match expressions will not be
// propogated to node affinity.
// CAUTION: Don't invoke WithAffinity func after calling this function
// It will overwrite MatchExpression
func (b *Builder) WithNodeAffinityMatchExpressions(
mExpressions []corev1.NodeSelectorRequirement) *Builder {
if len(mExpressions) == 0 {
return b
}

if b.podtemplatespec.Object.Spec.Affinity == nil {
b.podtemplatespec.Object.Spec.Affinity = &corev1.Affinity{}
}
if b.podtemplatespec.Object.Spec.Affinity.NodeAffinity == nil {
b.podtemplatespec.Object.Spec.Affinity.NodeAffinity = &corev1.NodeAffinity{}
}
if b.podtemplatespec.
Object.
Spec.
Affinity.
NodeAffinity.
RequiredDuringSchedulingIgnoredDuringExecution == nil {
b.podtemplatespec.
Object.
Spec.
Affinity.
NodeAffinity.
RequiredDuringSchedulingIgnoredDuringExecution = &corev1.NodeSelector{}
}

b.podtemplatespec.
Object.
Spec.
Affinity.
NodeAffinity.
RequiredDuringSchedulingIgnoredDuringExecution.
NodeSelectorTerms = append(b.podtemplatespec.
Object.
Spec.
Affinity.
NodeAffinity.
RequiredDuringSchedulingIgnoredDuringExecution.
NodeSelectorTerms,
corev1.NodeSelectorTerm{
MatchExpressions: mExpressions,
},
)
return b
}

// WithTolerationsByValue sets pod toleration.
// If provided tolerations argument is empty it does not complain.
func (b *Builder) WithTolerationsByValue(tolerations ...corev1.Toleration) *Builder {
Expand Down
7 changes: 7 additions & 0 deletions provisioner/env.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ const (
// NFSServerNamespace defines the namespace for nfs server objects
// Default value is menv.OpenEBSNamespace(operator namespace)
NFSServerNamespace menv.ENVKey = "OPENEBS_IO_NFS_SERVER_NS"

// NodeAffinityKey holds the env name representing Node affinity rules
NodeAffinityKey menv.ENVKey = "OPENEBS_IO_NFS_SERVER_NODE_AFFINITY"
)

var (
Expand Down Expand Up @@ -87,3 +90,7 @@ func getOpenEBSServiceAccountName() string {
func getNFSServerImage() string {
return menv.GetOrDefault(NFSServerImageKey, string(NFSServerDefaultImage))
}

func getNfsServerNodeAffinity() string {
return menv.Get(NodeAffinityKey)
}
1 change: 1 addition & 0 deletions provisioner/helper_kernel_nfs_server.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ func (p *Provisioner) createDeployment(nfsServerOpts *KernelNFSServerOptions) er
WithSecurityContext(&corev1.PodSecurityContext{
FSGroup: nfsServerOpts.fsGroup,
}).
WithNodeAffinityMatchExpressions(p.nodeAffinity.MatchExpressions).
WithContainerBuildersNew(
container.NewBuilder().
WithName("nfs-server").
Expand Down
Loading

2 comments on commit cbb6dc2

@mgrechukh
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shall rather be property of StorageClass (wrapped into annotation?), not the provisioner as whole.

@mynktl
Copy link
Contributor

@mynktl mynktl commented on cbb6dc2 Aug 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shall rather be the property of StorageClass (wrapped into annotation?), not the provisioner as a whole.

Hi @mgrechukh ,
Thanks for looking into nfs-provisioner.
Can you describe your use case?

Please sign in to comment.