--- alias: installation-guide-aws-storage description: "Configure storage using EFS or NFS file shares for your EKS cluster, involving IAM roles, CSI drivers, and S3 file" --- # Storage This section describes how to configure storage to serve as backend for volumes or storage classes in your EKS cluster. ## Cluster Storage We will present two different options for the cluster storage, allowing you to choose the one that best fits your needs. - **EFS** - Amazon Elastic File System - **NFS File Share** (backed-up by an S3 File Gateway) !!! info We recommend using EFS as it is a fully managed service and therefore does not require manual management or maintenance. ### Option 1. EFS - Amazon Elastic File System !!! note More information: - Before creating the storage class, verify that an Amazon EFS Container Storage Interface (CSI) Driver is deployed to the previously created Amazon EKS cluster. Also, verify that an AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider exists for the cluster. To check this topic, follow [this link](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). - **Create an IAM policy and role** 1. **Create an IAM policy** - Download the IAM policy that allows the CSI driver's service account to make calls to AWS APIs. This IAM policy is designed for the AWS EFS CSI (Container Storage Interface) driver, which allows Kubernetes to manage Amazon EFS (Elastic File System) volumes. This policy grants permissions needed for EFS operations from within a Kubernetes cluster. ```bash curl -o iam-policy-example.json https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json ``` - Create the policy based on the previously downloaded json. ```bash aws iam create-policy --policy-name AmazonEKS_EFS_CSI_Driver_Policy --policy-document file://iam-policy-example.json ``` 2. **Create an IAM role and attach the IAM policy to it.** - First, you need to find the cluster's OpenID Connect (OIDC) provider URL. This is possible through the following command: ```bash aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text ``` The output must be as follows: `https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE` - Create the IAM role, granting the Kubernetes service account the AssumeRoleWithWebIdentity action. This action allows the role to be assumed by an entity using a Web Identity Federation token (in this case, an OIDC token from Kubernetes). - Copy the following contents to a file named trust-policy. Replace `111122223333` with your account ID. Replace `EXAMPLED539D4633E53DE1B71EXAMPLE` and `region-code` with the values returned in the previous step. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa" } } } ] } ``` - Now, create the role. ```bash aws iam create-role --role-name AmazonEKS_EFS_CSI_DriverRole --assume-role-policy-document file://"trust-policy.json" ``` - Attach the IAM policy to the role with the following command. Replace **111122223333** with your account ID. ```bash aws iam attach-role-policy --policy-arn arn:aws:iam::111122223333:policy/AmazonEKS_EFS_CSI_Driver_Policy --role-name AmazonEKS_EFS_CSI_DriverRole ``` - Create a Kubernetes service account that is annotated with the ARN of the IAM role that you created. - Create a file with the following contents named efs-service-account.yaml and replace **111122223333** with your account ID. ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/name: aws-efs-csi-driver name: efs-csi-controller-sa namespace: kube-system annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/AmazonEKS_EFS_CSI_DriverRole ``` - Run the following command to create the storage class. The Kubernetes service account named efs-csi-controller-sa is annotated with the IAM role that you created named AmazonEKS_EFS_CSI_DriverRole. ```bash kubectl apply -f efs-service-account.yaml ``` - **Install the Amazon EFS driver** - There are three ways to install the driver: Helm, Manifest (private registry), or Manifest (public registry). We used the last option. 1. Download the manifest through the following command: ```bash kubectl kustomize "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=v1.4.0" > public-ecr-driver.yaml ``` 2. Edit the file and remove the lines that are responsible for the creation of the storage class named efs-csi-controller-sa because it was created in the previous step. After that, apply the manifest: ```bash kubectl apply -f public-ecr-driver.yaml ``` - **Create an Amazon EFS file system** - The Amazon EFS CSI driver supports the Amazon EFS access points, which are application-specific entry points into an Amazon EFS file system that make it easier to share a file system between multiple points. - Below are the different steps the you need to create an Amazon EFS file system for the previously created cluster. 1. Get the VPC (Virtual Private Cloud) ID where the cluster is in. ```bash vpc_id=$(aws eks describe-cluster --name my-cluster --query "cluster.resourcesVpcConfig.vpcId" --output text) ``` 2. Get the CIDR range for the cluster's VPC. ```bash cidr_range=$(aws ec2 describe-vpcs --vpc-ids $vpc_id --query "Vpcs[].CidrBlock" --output text) ``` 3. Create a security group with an inbound rule that allows inbound NFS traffic for the Amazon EFS mount points. - Create a security group. ```bash security_group_id=$(aws ec2 create-security-group --group-name MyEfsSecurityGroup --description "My EFS security group" --vpc-id $vpc_id --output text) ``` - Create an inbound rule that allows inbound NFS traffic from the CIDR for the cluster's VPC. ```bash aws ec2 authorize-security-group-ingress --group-id $security_group_id --protocol tcp --port 2049 --cidr $cidr_range ``` 4. Create an Amazon EFS file system for the previously created cluster. - Create a file system. ```bash file_system_id=$(aws efs create-file-system --region region-code --performance-mode generalPurpose --query 'FileSystemId' --output text) ``` - Create mount targets. First you need to find the ip address of the cluster nodes. This can be done through the following command: ```bash kubectl get nodes ``` The output needs to be as follows: ![nodes_cluster_info][nodes_cluster_info] - After that, determine the ids of the subnets in the VPC and which Availability Zone the subnet is in. ```bash aws ec2 describe-subnets --filters "Name=vpc-id,Values=$vpc_id" --query 'Subnets[*].{SubnetId: SubnetId,AvailabilityZone: AvailabilityZone,CidrBlock:CidrBlock}' --output table --region region-code ``` ![cluster_subnets][cluster_subnets] - Finally, add mount targets for the subnets that the nodes are in. Through this [link](https://www.ipaddressguide.com/cidr) you can find out in which CidrBlock the node's ip address is in. ```bash aws efs create-mount-target --file-system-id $file_system_id --subnet-id subnet-EXAMPLEe2ba886490 --security-groups $security_group_id --region region-code ``` - To ensure that everything was created correctly, you can deploy the sample application with dynamic provisioning presented at the end of [this guide](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html). - After that, you can create a storage class. Take what follows as an example: ![storage_class_example][storage_class_example] - Replace the fileSystemId value with the previously created file system. ### Option 2. NFS !!! note More information: and 1. **Create AWS Storage Gateway** 1. Open the [AWS Storage Gateway console](https://console.aws.amazon.com/storagegateway/home) and select **Create Gateway**. 1. **Gateway Settings** - For **Gateway name**, enter a name for the gateway. - For **Gateway time zone**, select the local time zone for the part of the world where you want to deploy the gateway. 1. **Gateway Options** - For **Gateway Type**, select **Amazon S3 File Gateway**. ![setup_gateway][setup_gateway] 1. **Platform Options** - For **Host platform**, select the platform on which the gateway is supposed to be deployed. On our side, we chose **Amazon EC2**. The gateway must be created in the same _Virtual private cloud (VPC)_ as the cluster, and the selected _VPC subnet_ can be one that is available in the VPC. - After filling out all the fields with the correct data, select **Launch Instance**. This step can take up to two minutes. Select **Next** when the previous task is completed. ![setup_gateway_platform_options][setup_gateway_platform_options] 1. **Connect to AWS** - For **Endpoint options** you can select **VPC hosted**, but you must have created all the inherent resources in advance. Otherwise, continue with **Publicly accessible**. - For **Gateway connection options**, continue with the default options. ![connect_to_aws][connect_to_aws] 1. **Review and activate** - In this step, you can edit the prefilled settings. After proceeding, all data cannot be changed. 1. **Configure gateway** - This is the last step and all configurations can be left with the default values. Select **Next** to finish the gateway configuration. ![configure_gateway][configure_gateway] 1. Now, on the **Gateway overview dashboard**, the resource is available. You may need to wait a few minutes for the status to turn green. At this point, the gateway is ready to use. With the gateway ready, it is time to create the file shares. Besides having the gateway up and running, a S3 bucket resource needs to be created for each file share. Buckets are containers for data stored in S3. ![gateway_overview][gateway_overview] 1. **S3 bucket creation** 1. To create this resource, go to [this dashboard](https://console.aws.amazon.com/s3/buckets) and select **Create bucket**. A page similar to the one below comes up. Fill out the **bucket name** and make sure that the AWS Region is the same as in the previous resources. All other configurations can be left with the default values. ![create_bucket][create_bucket] 1. If all goes well, the bucket is created and ready to use. ![buckets_dashboard][buckets_dashboard] 1. **File share creation** 1. **File share settings** - For **Gateway**, select the previously created gateway. - For **Amazon S3 location**, select **S3 bucket name** and fill out the corresponding field with its name. ![file_share_settings][file_share_settings] - By default, the file share name is prefilled with the S3 bucket name, but it can be changed to a different one. - For now, leave the other configurations with the default values and select **Next**. ![file_share_settings_2][file_share_settings_2] 1. **Amazon S3 storage settings** - Keep the configurations with the default values and proceed. ![amazon_s3_storage_settings][amazon_s3_storage_settings] 1. **File access settings** - Keep the configurations with the default values and proceed. ![file_access_settings][file_access_settings] ![file_access_settings_2][file_access_settings_2] 1. **Review and create** - Check that all settings are correct and proceed with creating the file share. 1. File share is ready to use when its status changes to **Available**. It may take a few minutes between creation and this status update. ![file_share_overview][file_share_overview] ## EKS and Database Shared Storage The EKS cluster and the SQL Server must have a shared storage space that allows both to read and write data, ensuring seamless access and interaction between them. [storage_class_example]: ../../images/aws/storage_class_example.png [setup_gateway]: ../../images/aws/setup_gateway.png [setup_gateway_platform_options]: ../../images/aws/setup_gateway_platform_options.png [connect_to_aws]: ../../images/aws/connect_to_aws.png [configure_gateway]: ../../images/aws/configure_gateway.png [gateway_overview]: ../../images/aws/gateway_overview.png [create_bucket]: ../../images/aws/create_bucket.png [buckets_dashboard]: ../../images/aws/buckets_dashboard.png [file_share_settings]: ../../images/aws/file_share_settings.png [file_share_settings_2]: ../../images/aws/file_share_settings_2.png [amazon_s3_storage_settings]: ../../images/aws/amazon_s3_storage_settings.png [file_access_settings]: ../../images/aws/file_access_settings.png [file_access_settings_2]: ../../images/aws/file_access_settings_2.png [file_share_overview]: ../../images/aws/file_share_overview.png [nodes_cluster_info]: ../../images/aws/nodes_cluster_info.png [cluster_subnets]: ../../images/aws/cluster_subnets.png