Configure your AWS Cloud Resources

To run Deeploy on AWS, you will need a basic set of cloud resources. In this article, we will guide you through the setup process on AWS. We recommend installing the AWS CLI and EKSCTL to complete the following steps successfully.

Architecture

The architecture typically consists of a separate development and production environment, as shown in the high-level architecture overview below. Annotations continue below the image.

1. Deeploy is designed to seamlessly integrate with version control systems and model frameworks for machine learning code deployments. Currently, we provide support for the following systems:
   • Gitlab
   • GitHub
   • Bitbucket
   • Azure DevOps
   • Hugging Face
2. We suggest using a managed Kubernetes cluster (Stateless): AWS EKS. For normal usage, Deeploy requires approximately 3 medium nodes; minimal requirements: 3 (v)CPU and 6 GB RAM). We suggest using auto scaling nodegroups.
3. We suggest using a managed PostgreSQL database server: AWS RDS for PostgreSQL. Every microservice has his own database. In the RDS database all data shared with or created with Deeploy will be stored. We advise to enable AWS RDS Storage Autoscaling.
4. We suggest using S3 storage to store ML model artefacts.
5. We suggest isolating single Deeploy environment installation within a separate AWS VPC (see Networking)
6. Deeploy requires an AWS KMS Symmetric key. This key is used to encrypt/decrypt Deeploy tokens in the database.
7. The easiest way to install Deeploy on the cloud resources is to use the AWS ECR images as provided by the AWS Marketplace.
8. Deeploy authenticates AWS resources using an IAM user or IAM role. In order to use roles with Deeploy uses AWS EKS Pod Identity Webhooks.

Data storage

Data is stored on the following AWS resources:

Set up the cloud resources

For a production environment, we strongly recommend managing your cloud resources as infrastructure as code, using tools such as AWS CloudFormation or Terraform.

AWS Elastic Kubernetes Server (EKS)

To set up AWS EKS for your Deeploy installation, follow the steps outlined in the AWS EKS guide. However, keep in mind the following specific considerations:

• Kubernetes version: we advise to use version 1.23+ This is currently the lowest supported version of Kubernetes.
• Deeploy can receive traffic on the Istio ingress gateway via a AWS loadbalancer
• Enable autoscaling for your managed node group: we advise to enable - Cluster autoscaling. Deeploy does check the health of the cluster using readiness probes.

The fastest way to setup a EKS cluster is to use eksctl:

eksctl create cluster -f <your-config>.yaml

Use the eksctl documentation for specific configuration instructions. Pay attention to the single or multi availability zone configuration as described in this article.

GPU support

For GPU support we recommend attaching an autoscaling node group with your preferred GPU node type that can scale to 0 to your EKS cluster. Alternatively use Karpenter as a flexible scheduler. To make sure no other pods will be scheduled on GPU nodes you can add the following taint to your node pool:

taints:
- key: nvidia.com/gpu
  value: present
  effect: NoSchedule

When specifying a GPU node for a Deeploy deployment, automatically the nvidia.com/gpu label will be applied. To make sure you can select the nodes that are scaled to 0 in Deeploy, add a list of node types to the values.yaml file when you install the Deeploy Helm chart.

AWS RDS for PostgreSQL

To set up AWS RDS for your Deeploy installation, follow the steps outlined in the AWS RDS for PostgreSQL guide. It is crucial to implement best practices for backing up and restoring data at any point in time, as described in this article. Additionally, keep in mind the following specific considerations:

• Create a separate user called Deeploy with read and write rights, and save the credentials to use in the Helm values.yaml file that you will create later on.
• Enable AWS RDS Storage Autoscaling to prevent manual interventions, to accommodate the amount of data increasing over time.
• Align the network configuration of the RDS database with the EKS cluster (same VPC and subnets). This will allow for data transfers over the internal AWS network.

AWS S3

To set up a AWS S3 bucket, use the AWS S3 guide. Additionally, keep in mind the following specific considerations

• We suggest using a single IAM user/role to access the S3 bucket from the EKS cluster, with the following minimal required IAM access policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
         "s3:ListBucket"
       ],
      "Resource": [
         "arn:aws:s3:::<YOUR-BUCKET-NAME>"
       ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
         "arn:aws:s3:::<YOUR-BUCKET-NAME>/*"
      ]
    }
  ]
}

• Create an access key and a secret key and save the credentials to use in the Helm values.yaml file that you will create later on.
• Create an S3 Gateway endpoint for your VPC. This will allow for data transfers over the internal AWS network.

AWS KMS

To set up a AWS KMS Symmetric Encryption key, use the AWS KMS Guide. Additionally, keep in mind the following specific considerations

• We suggest using a single IAM user/ to access the KMS from the EKS cluster using the following minimal required IAM access policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<YOUR-ACCOUNT-ID>:user/<YOUR-DEEPLOY-IAM-USERNAME>"
       },
       "Action": [
         "kms:Encrypt",
         "kms:Decrypt",
         "kms:ReEncrypt*",
         "kms:GenerateDataKey*",
         "kms:DescribeKey"
       ],
       "Resource": "*"
     },
     {
       "Effect": "Allow",
       "Principal": {
         "AWS": [
           "arn:aws:iam::<YOUR-ACCOUNT-ID>:root"
         ]
       },
       "Action": [
         "kms:*
       ],
       "Resource": "*"
     }
  ]
}

• Create an access key and a secret key and save the credentials to use in the Helm values.yaml file that you will create later on.

Networking

We suggest becoming familiar with AWS specific networking as described in this guide. Follow the best practices for networking for the EKS and RDS deployments. An example AWS architecture diagram including networking configuration:

keep in mind the following specific considerations:

• Deploy the RDS and EKS cluster within the same VPC but use different security groups to specifically whitelist the EKS nodegroup within the RDS database.

AWS resource health

We suggest setting up CloudWatch for monitoring and alerting related to resource health and credit balance. If you suspect an issue with your AWS Cloud Resources, check out the service health dashboard.