We have started using ECS and we are not quite sure if the behaviour we are experiencing is the correct one, and if it is, how to work around it.
We have setup a Beanstalk Docker Multicontainer environment which in the background uses ECS to manage everything, that has been working just fine. Yesterday, we created a standalone cluster in ECS "ecs-int", a task definition "ecs-int-task" and a service "ecs-int-service" associated to a load balance "ecs-int-lb" and we added one instance to the cluster.
When the service first ran, it worked fine and we were able to reach the docker service through the loadbalance. While we were playing with the instance security group that is associated to the cluster "ecs-int" we mistakenly removed the port rule where the container were running, and the health check started failing on the LB resulting it in draining the instance out from it. When it happened, for our surprise the service "ecs-int-service" and the task "ecs-int-task" automatically moved to the Beanstalk cluster and started running there creating an issue for our beanstalk app.
While setting up the service we setup the placement rule we set as "AZ Balanced Spread".
Should the service move around cluster? Shouldn't the service be attached only to the cluster it was originally created to? If this is the normal behaviour though, how can we set a rule so he service even if the instances for some reason fail the health check but to stick within the same cluster?
