deco - Operational
Previous page
Next page
deno - Operational
deno
VTEX - Operational
VTEX
Third Party: Amazon Web Services (AWS) → AWS ec2-sa-east-1 - Operational
Third Party: Cloudflare → Cloudflare Sites and Services → API - Operational
Third Party: Cloudflare → Cloudflare Sites and Services → CDN/Cache - Operational
Third Party: Cloudflare → Cloudflare Sites and Services → DNS Root Servers - Operational
Third Party: Cloudflare → Cloudflare Sites and Services → SSL Certificate Provisioning - Operational
Third Party: Cloudflare → Cloudflare Sites and Services → Workers - Operational
Third Party: Fly.io → AWS s3-sa-east-1 - Operational
Third Party: Fly.io → Application Hosting → GRU - São Paulo, Brazil - Operational
Third Party: Github → Actions - Operational
Third Party: Github → API Requests - Operational
Third Party: Github → Git Operations - Operational
Third Party: Github → Pull Requests - Operational
Third Party: Github → Webhooks - Operational
Third Party: Shopify → Shopify API & Mobile - Operational
Third Party: Shopify → Shopify Checkout - Operational
Third Party: Shopify → Shopify Point of Sale - Operational
Third Party: Supabase → Supabase Cloud → AWS ec2-sa-east-1 - Operational
Third Party: Discord → Voice → Brazil - Operational
Third Party: Fly.io → Edge Proxy → Routing - Operational
We changed our router table on k8s but didn't restart our nodes.
We rollback our changes to recover network communication.
No action taken by deco.This incident has been resolved.
~5% of requests resulting in 500.We are currently investigating this incident.
Admin panel was out for ~5 minutes.This does not affect any website in production.
We rolled back Deno on our infrastructure and rebuilt all unstable pods. Errors have now decreased to less than 0.001%.
An update of deno (1.45.2) increased pressure over memory so pods were not being able to sustain in our infra.
This issue is affecting only the sites that deployed the new state today.
No more issues relating with the 502 since 12:05PM (GMT-3)
We saw a increase on 502 errors on our infrastructure. Some pods were not scaling or deploying due to 429 errors on our infrastructure (a protective measure due to rate limit calls).
We relaxed our WAF rules to avoid 429 errors.
Pods are deploying as expected.
Jul 2024 to Sep 2024
Next
