A fix has been implemented and we are monitoring the results.
Posted Feb 06, 2020 - 14:04 PST
Update
The load balancers removing hosts sporadically was a symptom. We're continuing to investigate what appears to be regularly occuring spikes in cache gets at the :00 and :30 minute marks each hour.
Posted Feb 06, 2020 - 13:26 PST
Update
We're investigating what is causing the load balancer to remove hosts sporadically.
Posted Feb 06, 2020 - 09:56 PST
Investigating
We are investigating increased error rates and timeouts with supermarket.chef.io