Network Load Balancer Health Checks - From testing in staging, we noticed that reducing our network load balancer’s health-check intervals and thresholds helped reduce staging deploy time from ~9 to ~6 minutes.This further reduced service deploy time to 16-18 minutes. We opened support tickets with the ELB and Fargate teams to get those limits increased. Will try again later.”ĭue to Scribd’s high Fargate task volume, the number of start and stop requests we were making was too high for AWS’ default limits. “service production-web operations are being throttled.“service production-web operations are being throttled on elb.There were events with the following messages. ECS Throttling - During deployments, we investigated the “Events” tab of our main web ECS service.This reduced service refreshes to ~22 minutes. We looked at Datadog APM for the p99 of our longest running request and set the delay to 17s from the default of 300s. It did lead to a cleaner image but that wasn’t our initial goal.ĭeregistration Delay - This is a property on a load balancer’s target group that dictates how long a task stays in Draining state after it stops receiving requests. We reduced our compressed image size from ~900Mb to ~700Mb and it led to little to no improvement. However, unless you can drastically reduce your image size (think 1Gb to 100Mb), this will not lead to significant time reductions.
#Aws fargate startup time how to
Here are some changes we implemented -ĭocker Image Size Reduction - The first thing everyone thinks of when considering ECS Fargate speedups is how to reduce the image pull time since Fargate (unlike EC2) has no image caching. We use the default rolling deployment which stops and starts tasks to trigger a re-pulling of the freshly-uploaded ECR image.
Our starting implementation involved a few steps:
In this post, we’ll share how we improved our deployment speeds from ~40 minutes to less than 20 minutes. Scribd moved its monolith to AWS in April 2020 and as part of the migration, we had to design and implement a deployment pipeline for our new (and shiny) ECS Fargate infrastructure.
0 kommentar(er)
