Skip to main content

EKS Cluster and Create CSI Driver to store credentials in AWS Secrets Manager via SecretProviderClass


EKS Cluster | CSI Driver | SecretProviderClass | AWS Secrets Manager

Setup EKS Cluster and Manage Credentials at runtime using CSI driver using SecretProviderClass and Secrets Manager


Assuming you have Configured/Installed AWS CLI, EKSCTL, KUBECTL, HELM.


CSI Basic Information:

  1. CSI (Container Storage Interface) widely used as a Storage Technology. Created by Google | Mesosphere | Docker. 
  2. It has two two Plugins one runs on the Master Node (Centralized Controller Plugin) and another one on Worker Nodes (Decentralized headless Node Plugin). 
  3. CSI communication protocol is gRPC. 
  4. The communication between Container Orchestration to Controller Plugin (Master) and to Node Plugin (Worker Node) happens using gRPC. 
  5. CSI Drivers: vendor specific compiled into Kubernetes/openshift binaries. To use a CSI driver, a StorageClass needs to be assigned first. 
  6. The CSI driver is then set as the Provisioner for the Storage Class.
  7. CSI drivers provide three main services, which are: Identity | Controller and Node based.
  • Identity: For Plugins related.
  • Controller: For Cluster operations related.
  • Node: For Container and Local machine(s) operations related.
      


As shown above, this is how CSI Operates using the Remote Procedure Call by gRPC.

Now we will create the basic EKS Cluster using EKSCTL which in turn runs a Cloud Formation Stack to create the AWS resources.

Here is the basic ClusterConfig yaml file (my-cluster.yaml) to start the EKS Cluster named "basic-cluster".
------------------------------------------------------------------------------------------------------------------

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: basic-cluster
  region: ap-southeast-1
  version: "1.22"
managedNodeGroups:
  - name: mana-ng-pub
    instanceType: t3.medium
    minSize: 1
    desiredCapacity: 1
    maxSize: 1
    availabilityZones: ["ap-southeast-1a"]
    volumeSize: 20
    ssh: # Using existing EC2 Key pair
      allow: true
      publicKeyName: eks-demo-sg # had this key already, can replace this with yours
    tags:
      nodegroup-role: worker

------------------------------------------------------------------------------------------------------------------
Below is the "SecretProviderClass" yaml (demo-spc.yaml) file.
Note: Can skip this file shown below to give a basic idea about it. 
------------------------------------------------------------------------------------------------------------------

---
# apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 # deprecated
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secret-application
spec:
  provider: aws # for aws can use this else for any other Clouds can change this value
  secretObjects:
    - secretName: MySecret #application-api-key  # the k8s secret name
      type: Opaque
      data:
        - objectName: MySecret #secret-api-key  # reference the corresponding parameter
          key: api_key
  parameters:
    objects: |
      - objectName: "MySecret" #(secret-api-key, # the AWS secret)
        objectType: "secretsmanager"

------------------------------------------------------------------------------------------------------------------

1.a, Create the EKS Cluster using the below command and verify once done -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl create cluster -f my-cluster.yaml

2, Get EKS Cluster Information -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl get cluster
2022-05-11 23:59:36 [ℹ]  eksctl version 0.96.0
2022-05-11 23:59:36 [ℹ]  using region ap-southeast-1
NAME    REGION    EKSCTL CREATED
basic-cluster ap-southeast-1  True

3, Get EKS Cluster SVC running -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   47m

4, "Install the ASCP (AWS Secrets and Configuration Provider)" :
The ASCP is available on GitHub in the secrets-store-csi-provider-aws repository.
The repo also contains example YAML files for creating and mounting a secret.
You first install the Kubernetes Secrets Store CSI Driver, and then you install the ASCP.

a. To install the SecretsStore CSI Driver, run the following commands.
For full installation instructions, see Installation in the Secrets Store CSI Driver Book.
➜  AWS_ACCOUNT_DVO_EMAIL $ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
➜  AWS_ACCOUNT_DVO_EMAIL $ helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

To verify that Secrets Store CSI Driver has started, run:
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-k24z6   3/3     Running   0          31s

b. To install the ASCP, use the YAML file in GitHub repo deployment directory.
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
serviceaccount/csi-secrets-store-provider-aws created
clusterrole.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-rolebinding created
daemonset.apps/csi-secrets-store-provider-aws created

Note: The above command will create the following and the respective code for them is shown below -
- ServiceAccount
- ClusterRole
- ClusterRoleBinding
- DaemonSet
------------------------------------------------------------------------------------------------------------------
# https://kubernetes.io/docs/reference/access-authn-authz/rbac
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-aws
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-aws-cluster-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
subjects:
- kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: kube-system

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: csi-secrets-store-provider-aws
  labels:
    app: csi-secrets-store-provider-aws
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-aws
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-aws
    spec:
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: true
      containers:
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-6-gee95299-2022.04.14.21.07
          imagePullPolicy: Always
          args:
              - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
      volumes:
        - name: providervol
          hostPath:
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
      nodeSelector:
        kubernetes.io/os: linux
------------------------------------------------------------------------------------------------------------------

3, To create and mount a secret :
a, Set the AWS Region & name of your cluster as shell variables so you can use the min bash commands. For <REGION>, enter the AWS Region where your Amazon EKS cluster runs. For <CLUSTERNAME>, enter the name of your cluster.
➜  AWS_ACCOUNT_DVO_EMAIL $ export REGION=ap-southeast-1 && export CLUSTERNAME="basic-cluster"

b, Create a test secret:
➜  AWS_ACCOUNT_DVO_EMAIL $ aws --region "$REGION" secretsmanager  create-secret --name MySecret --secret-string '{"username":"foo", "password":"bar"}'
{
    "ARN": "arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8",
    "Name": "MySecret",
    "VersionId": "8e47ef05-4f60-499a-b842-90903c7082fd"
}

c, Create a Resource Policy for the Pod that Limits its Access to the Secret you created in the previous step.
For <SECRETARN>, use the ARN of the secret. Save the policy ARN in a shell variable.
➜  AWS_ACCOUNT_DVO_EMAIL $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"], "Resource": ["arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8"]}] }')

d, Create IAM OIDC Provider for Cluster if you don't already have one -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once
2022-05-12 00:19:43 [ℹ]  eksctl version 0.96.0
2022-05-12 00:19:43 [ℹ]  using region ap-southeast-1
2022-05-12 00:19:44 [ℹ]  will create IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"
2022-05-12 00:19:45 [✔]  created IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"

e, Create Service Account, Pod uses & associate the Resource Policy created in step3 with that service account.
For this tutorial, for the service account name, you use nginx-deployment-sa -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl create iamserviceaccount --name nginx-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

f, Create SecretProviderClass to specify which secret to mount in Pod.
The following command uses ExampleSecretProviderClass.yaml in the ASCP GitHub repo examples directory to mount the secret you created in step 1 -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml
Warning: secrets-store.csi.x-k8s.io/v1alpha1 is deprecated. 
Use "secrets-store.csi.x-k8s.io/v1" instead.
secretproviderclass.secrets-store.csi.x-k8s.io/nginx-deployment-aws-secrets created

------------------------------------------------------------------------------------------------------------------
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "MySecret" # can name as per your secret in AWS Secrets Manager
          objectType: "secretsmanager" # can change to AWS Secrets Parameter if using that.
------------------------------------------------------------------------------------------------------------------

g, Deploy your pod. The following command uses ExampleDeployment.yaml in the ASCP GitHub repo examples directory to mount the secret in /mnt/secrets-store in the pod -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml
service/nginx-deployment created
deployment.apps/nginx-deployment created

h, To verify Secret has been mounted properly, use the following command and confirm that your secret value appears -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it $(kubectl get pods | awk '/nginx-deployment/{print $1}' | head -1) cat /mnt/secrets-store/MySecret; echo
OR
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec nginx-deployment-76f76f6b68-5wgjn -- cat /mnt/secrets-store/MySecret; echo
{"username":"foo", "password":"bar"}

i, To Get inside the POD to check the secrets -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it nginx-deployment-76f76f6b68-5wgjn -- sh
and then run inside the POD -
# cat /mnt/secrets-store/MySecret
{"username":"foo", "password":"bar"}

4, Once all test and verified can delete the EKS Cluster and other things -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl delete cluster --name $CLUSTERNAME

---------
To get a shell access to the container running inside the application pod, all you have to do is:
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it --namespace <your namespace> <your pod name> -- sh
---------


Benefits of using CSI Driver:

1. Can store the secrets directly into the PODS using the CSI ➜ SecretProviderClass ➜ AWS Secrets Manager.
2. No need to have configurations for the application to pass the values.
3. Can retrieve the credentials, api-keys etc at run time from the mounted volumes in Deployment/Pods using the environment(s).
4. Can pass multiple secrets instead of single in the SecretProviderClass and can use them in application using the environment data.
5, And many more..


Ref:


Labels:
Location: 14 Siglap Link, Singapore 448869

Comments

Post a Comment

Popular posts from this blog

Defacing Sites via HTML Injections (XSS)

Defacing Sites via HTML Injections Defacing Sites via HTML Injections What Is HTML Injection: "HTML Injection" is called as the Virtual Defacement Technique and also known as the "XSS" Cross Site Scripting. It is a very common vulnerability found when searched for most of the domains. This kind of a Vulnerability allows an "Attacker" to Inject some code into the applications affected in order to bypass access to the "Website" or to Infect any particular Page in that "Website". HTML injections = Cross Site Scripting, It is a Security Vulnerability in most of the sites, that allows an Attacker to Inject HTML Code into the Web Pages that are viewed by other users. XSS Attacks are essentially code injection attacks into the various interpreters in the browser. These attacks can be carried out using HTML, JavaScript, VBScript, ActiveX, Flash and other clinet side Languages. Well crafted Malicious Code can even hep the ...
Post a Comment
Read more

Linux Systems Performance/Observability (BPF (bpfcc-tools), BCC Tools

  Linux System Performance/Observability Tools Linux Systems Performance/Observability (BPF (bpfcc-tools), BCC Tools Assuming you have Linux Server in place and have the required BPF aka BCC related packages installed on the system(s) for the required Linux distribution. BPF(eBPF) aka BCC Tools (bpfcc-tools) : BPF, which originally stood for Berkley Packet Filter is the dynamic tracing tools for Linux Systems.  BPF initially used for the speeding up for the tcpdump expressions and since then it has been know as the extended Berkley packet Filter (eBPF).  Its new uses are Tracing Tools where it provides programmability for the BPF Compiler Collection (BCC) and bpftrace front ends .   Example: execsnoop, biosnoop etc is a BCC Tool. When facing production performance crisis these such list of tools comes handy to trace and fix the issue. However, it requires certain KERNEL level config options to be enabled such as CONFIG_FTRACE, CONFIG_BPF. Profiling tools typically re...
Post a Comment
Read more