How do I scale in Docker Swarm Mode W Terraform Digital Ocean Load Balancing

Question

How can I perform automatic scaling based on metrics like CPU usage and Network usage?

For example create a new instance if my CPU usage is above 70% on all the nodes, and destroy an instance if the CPU usages falls below 30%. These would have to be sustained usages of course, like all nodes running at more than 70% CPU Usage for more than 5 minutes.

I did some research into this but its not very clear. For example Digital Ocean has a monitoring service which can send you an email if the CPU Usage is abnormally high (or Networking Load) based on a threshold you set. The problem is they don't have WebHook support yet so I can't send these signals to my application.

One idea I had was to set a special email for these notifications and check the email. Based on the email response I then create a modified terraform file and 'terraform apply' it to modify the infrastructure?

I'm not sure if this is good form either since the point of terraform is code as infrastructure, but if the infrastructure is designed to create new instances and destroy instances all the time based on CPU Usage or Network Usage I'm not sure how you convey that concept correctly in terraform?

Can anyone help me with this?

Answer 1

The solution you could build for Digital Ocean could work for all other Cloud providers as long as they provide an API for creating/destroying nodes.

So either you switch to AWS which has a built-in auto-scaling, either you build your own solution. It's a lot of work but it's not hard to set up. You need to have a full visibility on your cluster like a monitoring system that track all your cluster nodes from a node registry/discovery and then collect metrics and trigger alerts.

You may configure you alert system to run your Terraform job, ansible playbooks, trigger a Jenkins Job via HTTP api,... that will scale in/out your infrastructure based on the metrics.

I hope the above information would be helpful to you.