{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-18T02:03:04.239Z"
  },
  "data": {
    "id": "incident_statuspage_octopus-deploy_58sq0jtxxc1d",
    "slug": "octopus-deploy-some-octopus-cloud-customers-affected-by-parsing-issue-with-configmap-2025-01-22-0jtxxc1d",
    "title": "Some Octopus Cloud customers affected by parsing issue with ConfigMap on Kubernetes deployments",
    "summary": "Some Octopus Cloud customers affected by parsing issue with ConfigMap on Kubernetes deployments",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2025-01-22T17:47:53.974+00:00",
    "updatedAt": "2025-02-05T22:29:17.401+00:00",
    "resolvedAt": "2025-01-23T07:49:15.651+00:00",
    "provider": {
      "slug": "octopus-deploy",
      "name": "Octopus Deploy"
    },
    "affectedServices": [
      {
        "slug": "octopus-cloud",
        "name": "Octopus Cloud"
      }
    ],
    "links": {
      "html": "/incidents/octopus-deploy-some-octopus-cloud-customers-affected-by-parsing-issue-with-configmap-2025-01-22-0jtxxc1d",
      "api": "/api/v1/incidents/octopus-deploy-some-octopus-cloud-customers-affected-by-parsing-issue-with-configmap-2025-01-22-0jtxxc1d",
      "providerHtml": "/providers/octopus-deploy",
      "alerts": "https://outagedeck.com/account?stack=octopus-deploy&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Octopus Deploy reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_octopus_deploy_status",
      "kind": "official_api",
      "name": "Octopus Deploy Status",
      "checkedAt": "2026-11-18T02:00:12.619+00:00",
      "stale": false,
      "officialUrl": "https://status.octopus.com",
      "statusPageUrl": "https://status.octopus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_octopus-deploy_58sq0jtxxc1d_d07hvk0cmggn",
        "status": "investigating",
        "body": "We are currently investigating the issue",
        "createdAt": "2025-01-22T17:47:54.055+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_58sq0jtxxc1d_2lgp310gyjlk",
        "status": "identified",
        "body": "We have identified the issue with a recently released version of Octopus Server to cloud customers.\nA recent change has caused config maps with multi-line variables, created via the Kubernetes containers deployment step, to fail. Config maps created by the dedicated Kubernetes config map step are not affected, nor config maps created by Raw YAML or Helm steps.\nWe are in the process of fixing this and will update the public issue with the fixed version: https://github.com/OctopusDeploy/Issues/issues/9221.",
        "createdAt": "2025-01-23T01:42:58.636+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_58sq0jtxxc1d_n6dhrvxpw2l7",
        "status": "monitoring",
        "body": "A fix has been implemented and will be rolled out to affected cloud customers in their next maintenance window.",
        "createdAt": "2025-01-23T02:47:44.675+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_58sq0jtxxc1d_3vqj474j9wx9",
        "status": "resolved",
        "body": "We are currently rolling out the fix to Cloud customers. Instances will be upgraded in their next maintenance window. If you are affected by this issue and want to expedite the upgrade, please contact support@octopus.com",
        "createdAt": "2025-01-23T07:49:15.651+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_58sq0jtxxc1d_jcxj64y9ymnd",
        "status": "resolved",
        "body": "# Report and learnings: Configure and apply Kubernetes resources step error parsing configmap.yml\n\n###### Author: Kevin Tchang\n\n## Summary\n\nIn Octopus Server `2025.1.5751`, a bug caused the deployment of Kubernetes config maps containing multi-line variables, when created through the _Configure and apply Kubernetes resources_ step \\(built-in Kubernetes step for deploying containers\\), to fail. Config maps created using the dedicated Kubernetes config map step, as well as those generated with Raw YAML or Helm steps, were unaffected. This issue impacted Cloud customers, who experienced deployment failures that had previously been successful.\n\nThe bug was a regression caused by a change supporting manifest reporting for Kubernetes deployment steps, part of an upcoming feature. This change mistakenly caused line breaks in multi-line Octopus variable values to not be properly escaped when substituted into the config map's key-value pairs. The problem became apparent when customers had PEM certificates or JSON blobs that needed to be inserted into the config map. These were replaced verbatim in Calamari, leading to YAML formatting issues due to unescaped line breaks.\n\n## Background\n\nThe [_Configure and apply Kubernetes resources_](https://octopus.com/docs/kubernetes/steps/kubernetes-resources) step deploys a combination of Kubernetes Deployment, Service, and Ingress resources. It also allows the optional configuration and deployment of an associated Kubernetes ConfigMap and Secret for reference by the Deployment. \n\nTo support Rolling Update and Blue/Green deployment strategies, ConfigMap and Secret resources must have unique names for each Deployment version. These resources are assigned [computed names](https://octopus.com/docs/kubernetes/steps/kubernetes-resources?q=configmap#configmap-and-secret), which, by default, combine the resource name with the Octopus deployment ID, and are determined only at deployment time.\n\n## Incident timeline\n\n_\\(All dates and times below are shown in UTC\\)_\n\n##### **22/1/2025 – 7:31 \\(18:31 AEDT\\)**\n\nBegan receiving customer reports of an increase in failing Kubernetes deployments. These failures have been observed across various projects, with similar errors related to parsing config maps. Our support team worked with our customers to troubleshoot the reasons for the failures.\n\n##### **22/1/2025 – 10:58 Jan 22, 2025 \\(21:58 AEDT\\)**\n\nOur support team escalated the issue to our engineering teams.\n\n##### **22/1/2025 – 15:51 \\(2:51 AEDT\\)**\n\nOur internal incident response process was initiated.\n\n##### **22/1/2025 – 21:45 \\(8:45 AEDT\\)**\n\nOur engineers logged on and begin to identify the cause of the incident.\n\n##### **23/1/2025 – 1:42 \\(12:42 AEDT\\)**\n\nThe fix for the bug is merged, and our Status Page is updated to _Identified._\n\n##### **23/1/2025 – 2:47 \\(13:47 AEDT\\)**\n\nOur Status Page is updated to _Monitoring_ as we begin the process to expedite the release `2025.1.7128` of the fix to our affected Cloud customers.\n\n##### **23/1/2025 – 7:49 \\(18:49 AEDT\\)**\n\nStatus Page updated to _Resolved._\n\n## Technical details\n\nBefore the change to support manifest reporting, the Kubernetes container deployment step created associated Kubernetes config maps \\(and secrets\\) using the `kubectl create` command with the `--from-files` flag, where each config map key-value pair was sent to Calamari as an individual file. This process was updated to use the more standard `kubectl apply -f` method, where Octopus now sends a single YAML manifest to Calamari representing the config map. The YAML is generated from a config map resource that we build as an in-memory C# object.\n\nThe bug was introduced when the argument for the config map object used raw, unevaluated Octopus variable values. The issue wasn’t identified during testing because the deployment step involves two stages of variable substitution: the first on Octopus Server, and the second inside Calamari during deployment. The two substitution passes are necessary to support the use of computed names, ensuring that each deployment version has its own unique resources.\n\nThe change didn't account for multi-line strings as potential variables, causing newline characters to not be properly escaped before serialization. This issue occurred because encoding needs to happen on Octopus Server before the object is serialized into YAML. The second substitution in Calamari is direct on the YAML file. The bug was a regression, and the fix involved evaluating the values before serialization to ensure newline characters were handled correctly.\n\n## Remediation and next steps\n\nAt Octopus, we take deployment reliability very seriously. After this incident, we conducted a thorough review to identify areas where we can improve our processes, in light of the lessons learned.\n\nWe’ve identified a complex and unconventional area of the code—specifically script-based Kubernetes deployments—that requires further attention. Given the distinctive challenges these deployments present, we are committed to enhancing this area with additional tests to ensure better reliability.",
        "createdAt": "2025-02-05T22:04:40.401+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}