Outputs need to support sensitive values in OutputSpec

[James-Newman]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Currently, there does not appear to be a way to mark an output value as sensitive. When creating resources that generate their own sensitive values, the base Terraform module may set the output as sensitive, but in order to retrieve that value and have it stored in the workspace-outputs secret, it must be explicitly request as part of the workspace definition outputspec. The outputspec does not support a sensitive parameter, so when the HCL is rendered by the operator it can't mark the output value as sensitive, and so later versions (>0.14?) prevent such a plan from occurring.

This issue is different to #39, as it seems that the request there, while seemingly identical, is not solved by the provided fix of #80. Yes all outputs are now stored in a kubernetes secret, but we still can't access sensitive secrets.

Use case:

I want to create a simple AWS RDS instance, and have the password randomly generated and output so that my pods running in kubernetes can access the password for authentication. At present, I can output everything else that I need (endpoint, username etc.) except the password.

Recreating the issue

Given a simple RDS module that outputs a sensitive password value:

output "db_password" {
    value = aws_db_instance.main.password
    sensitive = true
}
resource "random_password" "password" {
  length           = 16
  special          = false
}
resource "aws_db_instance" "main" {
  allocated_storage    = "10"
  max_allocated_storage = "20"
  engine               = "postgres"
  engine_version       = "13"
  instance_class       = "db.t3.micro"
  name                 = "example"
  username             = "postgres"
  password             = random_password.password.result
  backup_retention_period = 5
  skip_final_snapshot  = true
  allow_major_version_upgrade = true
  auto_minor_version_upgrade  = true
  apply_immediately = true
}

This is then used within a workspace CRD in the following example

---
apiVersion: app.terraform.io/v1alpha1
kind: Workspace
metadata:
  name: myOrg-psql
spec:
  organization: myOrg
  secretsMountPath: "/tmp/secrets"
  module:
    source: "git@github.com:myOrg/terraform-postgres-db.git"
  outputs:
    - key: db_password
      moduleOutputName: db_password
  variables:
    - key: AWS_DEFAULT_REGION
      value: eu-west-2
      sensitive: false
      environmentVariable: true
    - key: AWS_ACCESS_KEY_ID
      sensitive: true
      environmentVariable: true
    - key: AWS_SECRET_ACCESS_KEY
      sensitive: true
      environmentVariable: true

This is then rendered in to HCL again by the operator and submitted to the Terraform Cloud API

terraform {
    backend "remote" {
        organization = "myOrg"
        workspaces {
            name = "namespace-myOrg-psql"
        }
    }
}
output "db_password" {
    value = module.operator.db_password
}
module "operator" {
    source = "git@github.com:myOrg/terraform-postgres-db.git"
}

When this is submitted to Terraform Cloud as part of the plan however, the following error is encountered when using any later terraform version (>0.14?) due to the sensitivity of the module.operator.db_password output.

The error is as follows:

Error: Output refers to sensitive values
on main.tf line 16:
	output "db_password" {
To reduce the risk of accidentally exporting sensitive data that was intended to be only internal, Terraform requires that any root module output containing sensitive data be explicitly marked as sensitive, to confirm your intent. If you do intend to export this data, annotate the output value as sensitive by adding the following argument: sensitive = true

Running this terraform manually without the terraform operator produces the same error. The error is not with the way that the HCL is delivered to the Terraform Cloud workspace, but in the way that the HCL is rendered in the first place.

We can determine that the cause of the problem is due to the operators inability to render sensitive outputs correctly, because we can take this rendered terraform, update it, and submit it manually.

terraform {
    backend "remote" {
        organization = "myOrg"
        workspaces {
            name = "namespace-myOrg-psql"
        }
    }
}
output "db_password" {
    value = module.operator.db_password
    sensitive = true
}
module "operator" {
    source = "git@github.com:myOrg/terraform-postgres-db.git"
}

Adding the supported sensitive = true parameter directly to the terraform allows this to have a successful plan and apply operation(s) locally.

Solution

Ultimately I believe that we need to support a sensitive parameter within the CRD spec (https://github.com/hashicorp/terraform-helm/blob/master/crds/app.terraform.io_workspaces_crd.yaml#L58) and all the supporting code within the operator to render the HCL correctly.

References

I believe that these are the initial attempts at solving this problem
#39
#80

Workspace CRD
https://github.com/hashicorp/terraform-helm/blob/master/crds/app.terraform.io_workspaces_crd.yaml#L58