Postmortem -
Read details
Feb 11, 18:38 UTC
Resolved -
This incident has been resolved.
Feb 10, 22:14 UTC
Update -
We are continuing to monitor for any further issues.
Feb 10, 22:09 UTC
Update -
Per AWS: [01:58 PM PST] We can confirm significant recovery for the DNS resolution errors for Cloudfront distributions.
Feb 10, 22:00 UTC
Monitoring -
Per AWS: [01:46 PM PST] We are seeing early signs of recovery, and continue to work toward full recovery.
JustiFi note: we have not seen the DNS resolution error internally in the past 10 minutes
Feb 10, 21:52 UTC
Update -
Per AWS:
[01:40 PM PST] We can confirm errors for DNS resolution for some Cloudfront distributions. During this time, customers may receive an NXDOMAIN response. Additionally, customers may also experience delayed propagation for changes to CloudFront distributions. We have identified the root cause of the issue and are actively working on multiple paths to resolving the errors. We have verified that our initial mitigation effort on a portion of the affected subsystem was successful, and we are actively working toward performing that mitigation across the fleet. We recommend customers continue to retry any failed requests while we work toward mitigation. A few services that use CloudFront distributions for delivering content may also be affected at this time.
Feb 10, 21:45 UTC
Update -
AWS is continuing to identify the impact of this service and now lists Cloudfront, Route 53 (dns), WAF (firewall) as impacted, all of which we use to serve our API.
Feb 10, 21:45 UTC
Update -
Per AWS: [01:15 PM PST] We are investigating DNS resolution failures for some specific Cloudfront distributions. We are actively investigating and will provide additional information in the next 30-60 minutes.
Feb 10, 21:18 UTC
Identified -
AWS has reported an issue impacting Cloudfront, which is the service which routes our API. We will update when we know more.
Feb 10, 21:17 UTC
Update -
This appears to be limited to our non-PCI AWS accounts, so our payments API is functional.
Feb 10, 21:04 UTC
Update -
It appears that around 1% of requests are having an issue with DNS, particularly when one of our internal services requests to another service.
Feb 10, 20:58 UTC
Investigating -
We saw a handful of 500 errors, and our API domain had DNS resolution errors starting at 20:27UTC. It appears to be intermitent. We are monitoring our error logs, and working on identifying an issues with our platform host provider AWS.
Feb 10, 20:42 UTC
