{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-18T01:08:10.781Z"
  },
  "data": {
    "id": "incident_statuspage_octopus-deploy_bf5gd9vgvfml",
    "slug": "octopus-deploy-some-octopus-cloud-customers-affected-by-missing-variable-substitution-2025-01-24-d9vgvfml",
    "title": "Some Octopus Cloud customers affected by missing variable substitution on Helm template sources",
    "summary": "Some Octopus Cloud customers affected by missing variable substitution on Helm template sources",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2025-01-24T01:35:41+00:00",
    "updatedAt": "2025-02-05T22:48:24.909+00:00",
    "resolvedAt": "2025-01-27T22:21:36+00:00",
    "provider": {
      "slug": "octopus-deploy",
      "name": "Octopus Deploy"
    },
    "affectedServices": [
      {
        "slug": "octopus-cloud",
        "name": "Octopus Cloud"
      }
    ],
    "links": {
      "html": "/incidents/octopus-deploy-some-octopus-cloud-customers-affected-by-missing-variable-substitution-2025-01-24-d9vgvfml",
      "api": "/api/v1/incidents/octopus-deploy-some-octopus-cloud-customers-affected-by-missing-variable-substitution-2025-01-24-d9vgvfml",
      "providerHtml": "/providers/octopus-deploy",
      "alerts": "https://outagedeck.com/account?stack=octopus-deploy&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Octopus Deploy reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_octopus_deploy_status",
      "kind": "official_api",
      "name": "Octopus Deploy Status",
      "checkedAt": "2026-11-18T01:00:13.651+00:00",
      "stale": false,
      "officialUrl": "https://status.octopus.com",
      "statusPageUrl": "https://status.octopus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_octopus-deploy_bf5gd9vgvfml_37t5cpfdjg9d",
        "status": "identified",
        "body": "We have had reports of and have identified an issue with variable substitution with Helm template sources. We are tracking the fix here: https://github.com/OctopusDeploy/Issues/issues/9224",
        "createdAt": "2025-01-24T01:35:41.312+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_bf5gd9vgvfml_2p31f7gws50j",
        "status": "monitoring",
        "body": "We have fixed the issue, but due to timing and other factors will not be rolling this out immediately. If you are affected by this issue, please get in touch with support@octopus.com who can manually update your instance.",
        "createdAt": "2025-01-24T08:58:36.815+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_bf5gd9vgvfml_d7jnftvz2l0t",
        "status": "resolved",
        "body": "This issue has been resolved and will be rolling out to Octopus Cloud customers in the next couple of days. If you are affected by this issue, please get in touch with support@octopus.com",
        "createdAt": "2025-01-27T22:21:36.566+00:00"
      },
      {
        "id": "update_statuspage_octopus-deploy_bf5gd9vgvfml_3z84j5yqj0ct",
        "status": "resolved",
        "body": "# Report and learnings: Missing variable substitution on Helm template sources\n\n###### Author: Kevin Tchang\n\n# Summary\n\nThe Octopus Server `2025.1.6849` update introduced a bug in the _Deploy a Helm Chart_ step, preventing variable substitution from working correctly in certain Helm template value sources. This led to Helm deployment failures for Cloud customers, which had previously succeeded without issue.\n\n# Background\n\n[Calamari](https://octopus.com/docs/octopus-rest-api/calamari) is a deployment tool used by Octopus to execute deployment tasks on target machines, such as extracting packages and running scripts. It supports the deployment of many built-in Octopus steps, including the _Deploy a Helm Chart_ step, which allows users to deploy Helm charts to Kubernetes clusters.\n\nWhen deploying Helm charts, Octopus allows users to pass values into the Helm release via **Helm Template Value Sources** \\(Helm TVS\\). These values can come from various sources, including charts, packages, Git repositories, key-values, or inline YAML. The Helm TVS are sent to Calamari during deployment to configure the Helm chart correctly.\n\n**Octopus variable substitution** allows dynamic replacement of placeholder variables within deployment configurations. This enables users to store sensitive data or environment-specific values as variables in Octopus and substitute them during deployment.\n\n## Incident timeline\n\n_\\(All dates and times below are shown in UTC\\)_\n\n##### 23/1/2025 – 18:49 \\(5:49 AEDT\\)\n\nWe receive the first reports of Helm deployments failing due to missing variable substitutions.\n\n##### 23/1/2025 – 23:39 \\(10:39 AEDT\\)\n\nOur support team escalated the issue to our engineering teams.\n\n##### 24/1/2025 – 1:35 \\(12:35 AEDT\\)\n\nOur internal incident response process was initiated.\n\n##### 24/1/2025 – 8:58 \\(19:58 AEDT\\)\n\nThe fix for the bug is merged `2025.1.7389`, and our Status Page is updated to _Monitoring_. However, due to the timing of a recent Octopus Server upgrade to .NET 9 and concerns about stability, the fix was not immediately rolled out to all Cloud customers. In the meantime, our support team provided assistance with manual upgrades.\n\n##### 27/1/2025 – 22:21 \\(9:21 AEDT\\)\n\nAfter the long weekend \\(due to a public holiday\\), the Status Page was updated to _Resolved_ once the fix was confirmed by customers.\n\n## Technical details\n\nHelm TVS are transmitted to Calamari as strings within a JSON array structure. Upon receiving these, Calamari parses the JSON array into their respective TVS types \\(such as package, chart, inline-YAML, etc.\\), which are then used by the `helm upgrade` command.\n\nThe bug was inadvertently introduced during a recent change aimed at eliminating the need for escaping quotes when using Octopus variable values in Helm TVS. Prior to this change, variable values with unescaped quotes caused parsing errors because the quotes were misinterpreted when processed from the JSON array structure.\n\nThe change modified the process so that variable substitution occurs after the JSON array is parsed into its respective TVS types, rather than before. However, this substitution was initially applied only to the **inline YAML TVS type**, and not the other four TVS types, which also could contain variables to substitute. This resulted in a regression, and the fix involved applying variable substitution to all TVS types and ensuring it was handled correctly wherever a variable could appear.\n\n## Remediation and next steps\n\nAt Octopus, ensuring deployment reliability is a top priority. Following this incident, we conducted a comprehensive review to identify areas for improvement. Given the flexibility and complexity of our product features, designing a robust testing process for all scenarios can be challenging. However, we’re taking proactive steps to improve how we test variable evaluation features.",
        "createdAt": "2025-02-05T22:30:53.229+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}