[Bug]: Adding new ou to the deployment_targets in aws_cloudformation_stack_set_instance fails

[michalz-rely]

Terraform Core Version

1.5.2

AWS Provider Version

5.15.0

Affected Resource(s)

aws_cloudformation_stack_set_instance

Expected Behavior

Modify the OUs that aws_cloudformation_stack_set_instance is applied to

Actual Behavior

No OU edits are possible

Relevant Error/Panic Output Snippet

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_cloudformation_stack_set_instance.eventbridge_us-east-1: Modifying... [id=eventbridge,ou-1,us-east-1]
╷
│ Error: updating CloudFormation StackSet Instance (eventbridge,ou-1,us-east-1): ValidationError: Organizational unit ou-2 not found in StackSet
│       status code: 400, request id: 55751d71-redacted
│ 
│   with aws_cloudformation_stack_set_instance.eventbridge_us-east-1,
│   on eventbridge_instances.autogen.tf line 4, in resource "aws_cloudformation_stack_set_instance" "eventbridge_us-east-1":
│    4: resource "aws_cloudformation_stack_set_instance" "eventbridge_us-east-1" {
│ 
╵
Releasing state lock. This may take a few moments...

Terraform Configuration Files

resource "aws_cloudformation_stack_set" "eventbridge" {
  provider         = aws.org_root
  name             = "eventbridge"
  permission_model = "SERVICE_MANAGED"
  auto_deployment {
    enabled                          = true
    retain_stacks_on_account_removal = false
  }
  lifecycle {
    ignore_changes = [
      administration_role_arn
    ]
  }
  template_body = file("${path.module}/eventbridge.yml")
}

resource "aws_cloudformation_stack_set_instance" "eventbridge_us-east-1" {
  provider       = aws.org_root
  region         = "us-east-1"
  stack_set_name = aws_cloudformation_stack_set.eventbridge.name
  operation_preferences {
    max_concurrent_percentage = 100
    failure_tolerance_percentage = 100
  }
  deployment_targets {
      organizational_unit_ids = ["ou-1","ou-2"]
  }
}

Steps to Reproduce

• Create aws_cloudformation_stack_set_instance and add organizational_unit as a deployment_target
• Add another OU as a deployment_target

Debug Output

aws_cloudformation_stack_set_instance.eventbridge_us-east-1: Modifying... [id=eventbridge,ou-1,us-east-1]
2023-10-05T17:58:16.814+0200 [INFO]  Starting apply for aws_cloudformation_stack_set_instance.eventbridge_us-east-1
2023-10-05T17:58:16.815+0200 [DEBUG] aws_cloudformation_stack_set_instance.eventbridge_us-east-1: applying the planned Update change
2023-10-05T17:58:16.816+0200 [DEBUG] provider.terraform-provider-aws_v5.15.0_x5: HTTP Request Sent: aws.operation=UpdateStackInstances aws.service=CloudFormation net.peer.name=cloudformation.us-west-2.amazonaws.com tf_mux_provider=*schema.GRPCProviderServer @module=aws aws.region=us-west-2 aws.sdk=aws-sdk-go http.method=POST http.request.header.x_amz_date=20231005T155816Z tf_resource_type=aws_cloudformation_stack_set_instance tf_rpc=ApplyResourceChange http.flavor=1.1 http.request.body="Action=UpdateStackInstances&CallAs=SELF&DeploymentTargets.OrganizationalUnitIds.member.1=ou-2&DeploymentTargets.OrganizationalUnitIds.member.2=ou-1&OperationId=terraform-20231005155816816000000001&OperationPreferences.FailureTolerancePercentage=100&OperationPreferences.MaxConcurrentPercentage=100&ParameterOverrides=&Regions.member.1=us-east-1&StackSetName=eventbridge&Version=2010-05-15
" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************JF67/20231005/us-west-2/cloudformation/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" http.url=https://cloudformation.us-west-2.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.15.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.334 (go1.20.7; darwin; arm64)" tf_req_id=13d0fcbd-2fbc-7d0f-7368-be3300f5477d @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.35/logger.go:96 http.request.header.x_amz_security_token=***** http.request_content_length=424 tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2023-10-05T17:58:16.816+0200
2023-10-05T17:58:19.630+0200 [DEBUG] provider.terraform-provider-aws_v5.15.0_x5: HTTP Response Received: http.response.header.x_amzn_requestid=13b0dca4-b521-4eaf-b412-ddbea369d4db http.status_code=400 aws.operation=UpdateStackInstances aws.sdk=aws-sdk-go http.response.header.content_type=text/xml http.response_content_length=337 @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.35/logger.go:144 @module=aws aws.region=us-west-2 http.response.header.date="Thu, 05 Oct 2023 15:58:19 GMT" tf_req_id=13d0fcbd-2fbc-7d0f-7368-be3300f5477d tf_rpc=ApplyResourceChange aws.service=CloudFormation http.response.header.connection=keep-alive tf_mux_provider=*schema.GRPCProviderServer tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_cloudformation_stack_set_instance http.duration=2814 http.response.body="<ErrorResponse xmlns="http://internal.amazon.com/coral/com.amazonaws.maestro.service.v20160713/">
  <Error>
    <Type>Sender</Type>
    <Code>ValidationError</Code>
    <Message>Organizational unit ou-2 not found in StackSet</Message>
  </Error>
  <RequestId>13b0dca4-b521-4eaf-b412-ddbea369d4db</RequestId>
</ErrorResponse>
" timestamp=2023-10-05T17:58:19.630+0200
2023-10-05T17:58:19.631+0200 [ERROR] provider.terraform-provider-aws_v5.15.0_x5: Response contains error diagnostic: diagnostic_detail= diagnostic_severity=ERROR tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=13d0fcbd-2fbc-7d0f-7368-be3300f5477d @caller=github.com/hashicorp/terraform-plugin-go@v0.18.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto tf_resource_type=aws_cloudformation_stack_set_instance tf_rpc=ApplyResourceChange diagnostic_summary="updating CloudFormation StackSet Instance (eventbridge,ou-1,us-east-1): ValidationError: Organizational unit ou-2 not found in StackSet
        status code: 400, request id: 13b0dca4-b521-4eaf-b412-ddbea369d4db" timestamp=2023-10-05T17:58:19.630+0200
2023-10-05T17:58:19.632+0200 [DEBUG] State storage *remote.State declined to persist a state snapshot
2023-10-05T17:58:19.632+0200 [ERROR] vertex "aws_cloudformation_stack_set_instance.eventbridge_us-east-1" error: updating CloudFormation StackSet Instance (eventbridge,ou-1,us-east-1): ValidationError: Organizational unit ou-2 not found in StackSet
        status code: 400, request id: 13b0dca4-b521-4eaf-b412-ddbea369d4db
2023-10-05T17:58:19.632+0200 [DEBUG] states/remote: state read serial is: 25; serial is: 25
2023-10-05T17:58:19.632+0200 [DEBUG] states/remote: state read lineage is: 8fb6b507-fb2e-a5a8-fa3e-c36e3a9c65af; lineage is: 8fb6b507-fb2e-a5a8-fa3e-c36e3a9c65af
╷
│ Error: updating CloudFormation StackSet Instance (eventbridge,ou-1,us-east-1): ValidationError: Organizational unit ou-2 not found in StackSet
│       status code: 400, request id: 13b0dca4-b521-4eaf-b412-ddbea369d4db
│ 
│   with aws_cloudformation_stack_set_instance.eventbridge_us-east-1,
│   on eventbridge_instances.autogen.tf line 4, in resource "aws_cloudformation_stack_set_instance" "eventbridge_us-east-1":
│    4: resource "aws_cloudformation_stack_set_instance" "eventbridge_us-east-1" {
│ 

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[evantlueck]

terraform 1.5.2
(v5.9.0) aws provider

Hello! I wasn't able to find a very code-concise solution, but this ended up working well for me. This is assuming your stackset related infrastructure code is inside a module with a aws_cloudformation_stack_set.stackset resource defined above.

variable "target_ous" {
  type        = list(string)
  description = "Where would you like to deploy the stackset?"
}

locals {
  deduped_ous = distinct(var.target_ous)
}

data "aws_organizations_organizational_unit_descendant_accounts" "targeted_accounts" {
  for_each = { for ou in local.deduped_ous : ou => ou }
  parent_id = each.key
}

locals {
  ou_descendants_map = { for ou, child_accounts in data.aws_organizations_organizational_unit_descendant_accounts.targeted_accounts : ou => [ for account in child_accounts.accounts : account.id ] }
  # Identify OUs whose account list is a subset of any other OU's account list
  subset_ous = [for ou_id, accounts in local.ou_descendants_map :
    ou_id if length([
      for other_ou_id, other_accounts in local.ou_descendants_map :
        other_ou_id if (other_ou_id != ou_id && length(setsubtract(accounts, other_accounts)) == 0)
    ]) > 0
  ]
  # list of OUs which are not a subset of any other OU's account list which should each have their own aws_cloudformation_stack_set_instance resource.
  final_ous = setsubtract(local.deduped_ous, local.subset_ous)
}

resource "aws_cloudformation_stack_set_instance" "instance" {
  for_each       = { for ou in local.final_ous : ou => [ou] }
  stack_set_name = aws_cloudformation_stack_set.stackset.name
  region         = data.aws_region.this.id
  retain_stack   = false
  deployment_targets {
    organizational_unit_ids = each.value
  }
  operation_preferences { # ref: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudformation_stack_set#operation_preferences-argument-reference
    failure_tolerance_percentage = 50
    max_concurrent_percentage    = 50
    region_concurrency_type      = "PARALLEL"
  }
}

From testing, I was able to replicate your issue as well. This solution ensures that every ou is evaluated against each other ou; and if an ou contains account targets which are a subset of another ou's account targets (i.e. a child of an already defined parent), the subset is filtered out of the deployment.

The only edge case error I found using this setup is if you have a parent ou and child ou (of that parent) targeted, the stackset has been deployed at least one time prior, and you remove the aforementioned parent ou without removing the child of that parent, then after that apply runs; on your next apply, will encounter an error similar to ValidationError: Organizational unit ou-xxxxxxxxxxx not found in StackSet mentioned above. Which, in this case, can be resolved by removing the aws_cloudformation_stack_set_instance resource associated with ou-xxxxxxxxxx from terraform state and then re-running a terraform apply.

But this edge case is completely avoidable by making sure to only target parent OUs to begin with.

In summary, the code above is expected for use within a module. It takes in a variable called target_ous (which would just be a list of generic ou id strings). It then parses that list, recursively checks for parent child relationships (removing any OUs passed to it which are seen to be children of a different OU which helps prevent terraform state issues which would arise from deploying multiple stackset instances to the same stackset whose child target accounts are shared by two OUs) and then creates a separate stackset instance resource for each unique OU.

Obviously, the issue with the provider remains. This doesn't fix the underlying logic issues with this bug thread. But it is a workaround to make sure your stackset use still LOGICALLY functions as you would expect. Therefore allowing you to create new OU targets, and remove OU targets, without terraform failing erroneously.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.58.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.