deco - Notice history

All systems operational

hosting-kubernets - Operational

100% - uptime
Nov 2024 · 99.96%Dec · 99.89%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

hosting-denodeploy - Operational

admin - Operational

100% - uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 100.0%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

VTEX - Operational

Third Party: Amazon Web Services (AWS) → AWS ec2-sa-east-1 - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → API - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → CDN/Cache - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → DNS Root Servers - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → SSL Certificate Provisioning - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → Workers - Operational

Third Party: Fly.io → AWS s3-sa-east-1 - Operational

Third Party: Fly.io → Application Hosting → GRU - São Paulo, Brazil - Operational

Third Party: Github → Actions - Operational

Third Party: Github → API Requests - Operational

Third Party: Github → Git Operations - Operational

Third Party: Github → Pull Requests - Operational

Third Party: Github → Webhooks - Operational

Third Party: Shopify → Shopify API & Mobile - Operational

Third Party: Shopify → Shopify Checkout - Operational

Third Party: Shopify → Shopify Point of Sale - Operational

Third Party: Supabase → Supabase Cloud → AWS ec2-sa-east-1 - Operational

Third Party: Discord → Voice → Brazil - Operational

Third Party: Supabase → Auth - Operational

Third Party: Supabase → Storage - Operational

Notice history

Jan 2025

Unable to upload assets to deco admin
  • Resolved
    Resolved

    No errors logged since 14:45:11 (UTC).

    We are closing this issue.

  • Monitoring
    Monitoring

    We are pushing a fix and monitoring it.

  • Identified
    Identified

    An update on our aws dependencies broke the submit asset feature. We are investigating solutions.

  • Investigating
    Investigating
    We are currently investigating this incident.
Unable to create/use some environments
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We deployed a solution. Environments may need a reset to go back to work.

  • Identified
    Identified

    Sites that received a browserslist update (dependency) are unable to run, due to lack of permissions to access filesystem.

    We are deploying some solutions to mitigate this issue.

    Rollback and lock to browserslist 4.24.3 may fix this problem.

  • Investigating
    Investigating
    We are currently investigating this incident.

Dec 2024

K8s infrastructure is not able to create/run environments
  • Resolved
    Resolved

    We reversed our node policies, and environments are now going back as expected.

    This incident has been resolved.

  • Identified
    Identified

    Some pods were not able to claim a PV due to some policies on our instance role.

  • Investigating
    Investigating

    We are currently investigating this incident.

    Previews / admin environments are not able to run.

Elevated 502 Errors due to third-party Infrastructure Issues
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Partial Increase in 5xx Errors
  • Identified
    Identified

    The thirdparty provider is aware and acting

  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are currently observing an elevated number of 5xx errors on a subset of our sites. Preliminary investigation suggests these errors are resulting from issues within a third-party infrastructure component that some of our services depend on.

DNS propagation issues
  • Resolved
    Resolved

    We moved our sites to a new DNS, and already recovered our decocdn.com.

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Update
    Update

    Our decocdn.com domain is not propagating our address.

    We are migrating sites from that domain and in contact with our dns registrar to also fix that issue for our current domain.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.

Nov 2024

Outage of some sites (k8s infra)
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We disable one layer of cache, and are monitoring sites going back.

  • Identified
    Identified

    One layer of our tiered cache strategy had a bottleneck on connections/usage.

  • Investigating
    Investigating
    We are currently investigating this incident.

Nov 2024 to Jan 2025

Next