aws_route53_vpc_association_authorization Destroys with Errors when Multiple VPCs are Associated

[rschwartz-tpn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.29

• provider.aws v3.11.0
• provider.external v1.2.0
• provider.local v1.4.0

Affected Resource(s)

• aws_route53_vpc_association_authorization

Terraform Configuration Files

# local.region_zone_id_map = [
#   "af-south-1_vpc-111_Z02887491MAG9UV3HPBSQ",
#   "us-east-1_vpc-222_Z02887491MAG9UV3HPBSQ",
#   "us-west-1_vpc-333_Z02887491MAG9UV3HPBSQ"
#  ]
resource "aws_route53_vpc_association_authorization" "zone_auth" {
  provider = aws.spoke

  for_each = toset(local.region_zone_id_map)

  zone_id    = element(split("_", each.key), 2) 
  vpc_id     = element(split("_", each.key), 1) 
  vpc_region = element(split("_", each.key), 0)
}

Debug Output

Initial Plan:
=============

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # module.authorize_domain_spoke.aws_route53_vpc_association_authorization.zone_auth["af-south-1_vpc-111_Z02887491MAG9UV3HPBSQ"] will be destroyed
  - resource "aws_route53_vpc_association_authorization" "zone_auth" {
      - id         = "Z02887491MAG9UV3HPBSQ:vpc-111" -> null
      - vpc_id     = "vpc-111" -> null
      - vpc_region = "af-south-1" -> null
      - zone_id    = "Z02887491MAG9UV3HPBSQ" -> null
    }

  # module.authorize_domain_spoke.aws_route53_vpc_association_authorization.zone_auth["us-east-1_vpc-222_Z02887491MAG9UV3HPBSQ"] will be destroyed
  - resource "aws_route53_vpc_association_authorization" "zone_auth" {
      - id         = "Z02887491MAG9UV3HPBSQ:vpc-222" -> null
      - vpc_id     = "vpc-222" -> null
      - vpc_region = "us-east-1" -> null
      - zone_id    = "Z02887491MAG9UV3HPBSQ" -> null
    }

  # module.authorize_domain_spoke.aws_route53_vpc_association_authorization.zone_auth["us-west-1_vpc-333_Z02887491MAG9UV3HPBSQ"] will be destroyed
  - resource "aws_route53_vpc_association_authorization" "zone_auth" {
      - id         = "Z02887491MAG9UV3HPBSQ:vpc-333" -> null
      - vpc_id     = "vpc-333" -> null
      - vpc_region = "us-west-1" -> null
      - zone_id    = "Z02887491MAG9UV3HPBSQ" -> null
    }

Plan: 0 to add, 0 to change, 3 to destroy.

Apply:
======

Error: Error deleting Route53 VPC Association Authorization: ConcurrentModification: A conflicting modification to the authorizations in place for Z02887491MAG9UV3HPBSQ occurred. Please retry.
        status code: 400, request id: 29c5b93b-86cf-405b-9f67-7678d45d9f78


Plan 2:
=======

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # module.authorize_domain_spoke.aws_route53_vpc_association_authorization.zone_auth["us-east-1_vpc-222_Z02887491MAG9UV3HPBSQ"] will be destroyed
  - resource "aws_route53_vpc_association_authorization" "zone_auth" {
      - id         = "Z02887491MAG9UV3HPBSQ:vpc-222" -> null
      - vpc_id     = "vpc-222" -> null
      - vpc_region = "us-east-1" -> null
      - zone_id    = "Z02887491MAG9UV3HPBSQ" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.


Apply 2:
========

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Expected Behavior

I would have expected the initial apply to destroy all resources without error.

Steps to Reproduce

1. With an existing PHZ with 4 VPCs associated and working, delete the PHZ via AWS Console
2. terraform apply that runs the above code

Important Factoids

I've experienced no problems when creating the resource - which adds 3 VPCs to an existing PHZ that already has 1 VPC associated to it.

I have run this twice (on two different test zones I created and deleted) - the first time, two VPCs did not destroy correctly and I had to run apply two more times for all the resources to get destroyed. The plan/apply listing above was for the second PHZ test.

Not sure if this is a factor, but the VPCs we are adding and removing are in different Regions, with the exception of us-east-1 where there are 2 VPC associations - of which we are removing one.

• #0000

[anGie44]

Hi @rschwartz-tpn, thank you for raising this issue. Making an initial pass at the error behavior you're seeing and given the AWS API indicates the error is retryable, I believe this will have to be addressed w/in the resource code by adding retry handling within the resource's Delete CRUD operation; currently we only call DeleteVPCAssociationAuthorization once and return on error.

[tmatilai]

The same error seems to happen sometimes also when creating multiple authorizations. Another apply succeeds, so retry really should help.

[YakDriver]

Related #18752

[boxrick]

So I have come across this error, using -parallelism=4 reduces the errors. I am trying to change 9 authorisations at the same time and get 4 errors. Adding the parallelism reduces this to 2 errors.

[briantist]

Just ran into this with creates; very annoying since the combination of vpcs and rules to be shared in my case results in thousands of objects, so even just refreshing the state to do another apply takes a very long time.

[github-actions]

This functionality has been released in v5.1.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.