[Bug]: Error when importing aws_glue_partition

[awsandy]

Terraform Core Version

1.9.3

AWS Provider Version

Affected Resource(s)

aws_glue_partition

When importing a partition the import throws an unexpected error

│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}

Expected Behavior

Resource should be imported without this error

Actual Behavior

The resource is not imported and you get this error:

│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}

Relevant Error/Panic Output Snippet

│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}

Terraform Configuration Files

import {
  to = aws_glue_partition.p-566972129213__sechub__brewed_sechub2_recipe_output__sechub2_29Mar2021_1617005427301
  id = "566972129213:sechub:brewed_sechub2_recipe_output:sechub2_29Mar2021_1617005427301"
}

Steps to Reproduce

import.tf:

import {
  to = aws_glue_partition.p-566972129213__sechub__brewed_sechub2_recipe_output__sechub2_29Mar2021_1617005427301
  id = "566972129213:sechub:brewed_sechub2_recipe_output:sechub2_29Mar2021_1617005427301"
}

terraform plan -generate-config-out=resources.out

│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}

& the resource is not imported.

Here is the partition configuration:

awsandy@bcd0749e47ec my-aws-samples % aws glue get-partitions --database-name sechub --table-name brewed_sechub2_recipe_output
{
    "Partitions": [
        {
            "Values": [
                "sechub2_29Mar2021_1617005427301"
            ],
            "DatabaseName": "sechub",
            "TableName": "brewed_sechub2_recipe_output",
            "CreationTime": "2021-03-29T11:18:50+01:00",
            "LastAccessTime": "2021-03-29T11:18:50+01:00",
            "StorageDescriptor": {
                "Columns": [
                    {
                        "Name": "awsaccountid",
                        "Type": "string"
                    },
                    {
                        "Name": "compliance_status",
                        "Type": "string"
                    },
                    {
                        "Name": "compliance_statusreasons",
                        "Type": "array<struct<Description:string,ReasonCode:string>>"
                    },
                    {
                        "Name": "createdat",
                        "Type": "string"
                    },
                    {
                        "Name": "description",
                        "Type": "string"
                    },
                    {
                        "Name": "firstobservedat",
                        "Type": "string"
                    },
                    {
                        "Name": "generatorid",
                        "Type": "string"
                    },
                    {
                        "Name": "id",
                        "Type": "string"
                    },
                    {
                        "Name": "lastobservedat",
                        "Type": "string"
                    },
                    {
                        "Name": "productarn",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_recommendationurl",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_relatedawsresources:0/name",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_relatedawsresources:0/type",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_relatedawsresources:1/name",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_relatedawsresources:1/type",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_ruleid",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_standardscontrolarn",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_standardsguidearn",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_standardsguidesubscriptionarn",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_aws/securityhub/companyname",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_aws/securityhub/findingid",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_aws/securityhub/productname",
                        "Type": "string"
                    },
                    {
                        "Name": "productfields_aws/securityhub/annotation",
                        "Type": "string"
                    },
                    {
                        "Name": "recordstate",
                        "Type": "string"
                    },
                    {
                        "Name": "remediation_recommendation",
                        "Type": "struct<Text:string,Url:string>"
                    },
                    {
                        "Name": "awsresources",
                        "Type": "struct<Details:struct<AwsCloudTrailTrail:struct<CloudWatchLogsLogGroupArn:string,CloudWatchLogsRoleArn:string,HasCustomEventSelectors:boolean,HomeRegion:string,IncludeGlobalServiceEvents:boolean,IsMultiRegionTrail:boolean,IsOrganizationTrail:boolean,KmsKeyId:string,LogFileValidationEnabled:boolean,Name:string,S3BucketName:string,S3KeyPrefix:string>,AwsEc2SecurityGroup:struct<GroupId:string,GroupName:string,IpPermissions:array<struct<FromPort:bigint,IpProtocol:string,IpRanges:array<struct<CidrIp:string>>,Ipv6Ranges:array<struct<CidrIpv6:string>>,ToPort:bigint,UserIdGroupPairs:array<struct<GroupId:string,UserId:string>>>>,IpPermissionsEgress:array<struct<FromPort:bigint,IpProtocol:string,IpRanges:array<struct<CidrIp:string>>,Ipv6Ranges:array<struct<CidrIpv6:string>>,ToPort:bigint,UserIdGroupPairs:array<struct<GroupId:string,UserId:string>>>>,OwnerId:string,VpcId:string>,AwsEc2Vpc:struct<CidrBlockAssociationSet:array<struct<AssociationId:string,CidrBlock:string,CidrBlockState:string>>,DhcpOptionsId:string,State:string>,AwsIamPolicy:struct<AttachmentCount:bigint,CreateDate:string,DefaultVersionId:string,Description:string,IsAttachable:boolean,Path:string,PermissionsBoundaryUsageCount:bigint,PolicyId:string,PolicyName:string,PolicyVersionList:array<struct<CreateDate:string,IsDefaultVersion:boolean,VersionId:string>>,UpdateDate:string>,AwsIamUser:struct<AttachedManagedPolicies:array<struct<PolicyArn:string,PolicyName:string>>,CreateDate:string,GroupList:array<string>,Path:string,UserId:string,UserName:string,UserPolicyList:array<struct<PolicyName:string>>>,AwsKmsKey:struct<AWSAccountId:string,CreationDate:bigint,Description:string,KeyId:string,KeyManager:string,KeyState:string,Origin:string>>,Id:string,Partition:string,Region:string,Type:string>"
                    },
                    {
                        "Name": "schemaversion",
                        "Type": "string"
                    },
                    {
                        "Name": "severity_label",
                        "Type": "string"
                    },
                    {
                        "Name": "severity_normalized",
                        "Type": "bigint"
                    },
                    {
                        "Name": "severity_original",
                        "Type": "string"
                    },
                    {
                        "Name": "severity_product",
                        "Type": "bigint"
                    },
                    {
                        "Name": "title",
                        "Type": "string"
                    },
                    {
                        "Name": "types",
                        "Type": "array<string>"
                    },
                    {
                        "Name": "updatedat",
                        "Type": "string"
                    },
                    {
                        "Name": "workflow_status",
                        "Type": "string"
                    },
                    {
                        "Name": "workflowstate",
                        "Type": "string"
                    }
                ],
                "Location": "s3://at-hmrc1/sechub2-recipe-output/sechub2_29Mar2021_1617005427301/",
                "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
                "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
                "Compressed": false,
                "NumberOfBuckets": -1,
                "SerdeInfo": {
                    "SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
                    "Parameters": {
                        "serialization.format": "1"
                    }
                },
                "BucketColumns": [],
                "SortColumns": [],
                "Parameters": {
                    "sizeKey": "570207",
                    "objectCount": "1",
                    "recordCount": "869",
                    "averageRecordSize": "617",
                    "compressionType": "none",
                    "classification": "parquet",
                    "typeOfData": "file"
                },
                "StoredAsSubDirectories": false
            },
            "Parameters": {},
            "CatalogId": "566972129213"
        }
    ]
}

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[awsandy]

Provider version is 5.63.0

[atlesp]

We also see the same error when we just now try to plan an existing project

Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}

The aws_glue_partition code:

resource "aws_glue_partition" "gold_findings_partitions" {
  for_each         = toset(local.findings_table_partitions)
  database_name    = aws_glue_catalog_database.gold.name
  table_name       = aws_glue_catalog_table.findings_gold.name
  partition_values = ["${each.value}"]
  storage_descriptor {
    location      = "s3://${module.s3_bucket_gold.name}/tables/${aws_glue_catalog_table.findings_gold.name}/resource_type=${each.value}"
    input_format  = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"
    ser_de_info {
      serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
    }

    parameters = {
      "classification" = "Parquet"
      "typeOfData" : "file"
      "compressionType" : "snappy"
    }

    stored_as_sub_directories = false
  }
}

It seems like it is related to this change #37891 & #34018

[robotparty]

I am also having this problem when specifying glue partitions using aws_glue_partition although I'm not importing.
I've not specified additional_locations anywhere in my config so not sure why it's thinking I did...

Terraform v1.9.4

AWS provider:

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.70.0"
  constraints = "~> 5.0, ~> 5.66, >= 5.70.0"

TF partition config:

resource "aws_glue_partition" "cloudtrail" {
  for_each      = local.partitions
  database_name = aws_glue_catalog_database.current.name
  table_name    = aws_glue_catalog_table.cloudtrail.name

  partition_values = [each.value.year, each.value.month]

  storage_descriptor {
    location      = "${local.cloudtrail_base_location}/${each.value.year}/${each.value.month}"
    input_format  = "com.amazon.emr.cloudtrail.CloudTrailInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"

    ser_de_info {
      serialization_library = "com.amazon.emr.hive.serde.CloudTrailSerde"
    }
  }
}

I'm creating. The partitions are created somehow but my TF plan says it fails:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}
│ 
│   with module.quicksight_usage_secondary.aws_glue_partition.cloudtrail["2023-4"],
│   on modules/quicksight_usage/regional/quicksight_usage_glue_catalog.tf line 386, in resource "aws_glue_partition" "cloudtrail":
│  386: resource "aws_glue_partition" "cloudtrail" {
│ 
╵
╷
│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}
│ 
│   with module.quicksight_usage_secondary.aws_glue_partition.cloudtrail["2023-2"],
│   on modules/quicksight_usage/regional/quicksight_usage_glue_catalog.tf line 386, in resource "aws_glue_partition" "cloudtrail":
│  386: resource "aws_glue_partition" "cloudtrail" {
│ 
╵
╷
...

One failure per partition.
Looks like the problem might be in these tests? That trailing .0 looks sus to me given the error...

Any news on this issue?

[robotparty]

Furthermore, the docs for the aws_glue_partition resource don't show an additional location attribute nor will my TF validate if I add one 😩 https://registry.terraform.io/providers/-/aws/latest/docs/resources/glue_partition

[robotparty]

The terraform lsp doesn't recognize the attribute either... it IS a property of aws_glue_catalog_table so why is it producing errors in the aws_glue_partition resource?

[robotparty]

The partition resources are tainted due to this error:

    },
        {
          "index_key": "2023-2",
          "status": "tainted",
          "schema_version": 0,
          "attributes": {
            "catalog_id": "012345678910",
            "creation_time": "2024-10-11T15:32:28Z",
            "database_name": "quicksight-usage",
            "id": "012345678910:usage:usage-cloudtrail:2023#2",
            "last_accessed_time": null,
            "last_analyzed_time": null,
            "parameters": null,
            "partition_values": [
              "2023",
              "2"
            ],```

[robotparty]

Still receiving this error in AWS Provider v5.80.0:

╷
│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}
│ 
│ 

[robotparty]

Glue partition properties in AWS (redacted):

{
  "Values": [
    "2024",
    "04"
  ],
  "DatabaseName": "quicksight-usage",
  "TableName": "quicksight-usage-cloudtrail",
  "CreationTime": "2024-10-11T22:48:33.000Z",
  "StorageDescriptor": {
    "Columns": [],
    "Location": "s3://my-bucket/AWSLogs/o-12345678a/12345678910/CloudTrail/us-east-1//2024/04/",
    "InputFormat": "com.amazon.emr.cloudtrail.CloudTrailInputFormat",
    "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
    "Compressed": false,
    "NumberOfBuckets": 0,
    "SerdeInfo": {
      "SerializationLibrary": "com.amazon.emr.hive.serde.CloudTrailSerde"
    },
    "SortColumns": [],
    "StoredAsSubDirectories": false
  },
  "CatalogId": "12345678910"
}

[timschill]

Facing the same error while creating a partition, the same input work fine while working with boto3 or the AWS CLI. AWS provider 5.83.1

The problem seems to originate in the fact that the flattenStorageDescriptor function (called by partition.go) returns a map containing a key named "additional_locations", yet in partition.go’s schema for storage_descriptor, there is no "additional_locations" field

[SpComb]

The same Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"} occurs when upgrading an existing terraform environment with a aws_glue_partition resource from terraform-provider-aws 4.67.0 -> 5.84.0, without involving any terraform import

╷
│ Error: setting storage_descriptor: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}
│ 
│   with module.aws_athena_alb_access_logs.aws_glue_partition.alb_access_logs["xxx"],
│   on ../../terraform-modules/aws_athena_alb_access_logs/glue.tf line 180, in resource "aws_glue_partition" "alb_access_logs":
│  180: resource "aws_glue_partition" "alb_access_logs" {
│ 
╵

[SpComb]

Using TF_LOG=debug shows the following, note tf_rpc=ReadResource:

2025-02-03T10:21:51.441+0200 [ERROR] provider.terraform-provider-aws_v5.84.0_x5: [ERROR] setting state: Invalid address to set: []string{"storage_descriptor", "0", "additional_locations"}
2025-02-03T10:21:51.443+0200 [ERROR] provider.terraform-provider-aws_v5.84.0_x5: Response contains error diagnostic: diagnostic_summary="setting storage_descriptor: Invalid address to set: []string{\"storage_descriptor\", \"0\", \"additional_locations\"}" @caller=github.com/hashicorp/terraform-plugin-go@v0.25.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_severity=ERROR tf_req_id=ea629bc8-82a6-82aa-b800-56693711b92c tf_resource_type=aws_glue_partition diagnostic_detail="" tf_proto_version=5.7 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ReadResource timestamp="2025-02-03T10:21:51.443+0200"

It looks like nikhil-goenka@bc86ca4#diff-5e4d893235d1f275a3d26ed5f4c5c22f4d2ef57dfbeb30bb29a4c6b15cca9121R915 modified the flattenStorageDescriptor() function in catalog_table.go to return a additional_locations key in the []map[string]interface{} return value.

https://github.com/nikhil-goenka/terraform-provider-aws/blob/c428d2cb510d33fcb603d7e6e22d1370416f642a/internal/service/glue/partition.go#L267-L269 however, the aws_glue_partition resource in partition.go uses the same flattenStorageDescriptor() function from catalog_table.go in its resource read function to set the storage_descriptor data, which fails because of the additional_locations key that is not defined in the aws_glue_partition resource schema.

I think this is thus a regression in #37891 -> terraform-provider-aws 5.55 that breaks all use of aws_glue_partition. Ping @nikhil-goenka ?

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.88.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!