[Bug]: aws_s3_bucket_lifecycle_configuration should honor the new explicit x-amz-transition-default-minimum-object-size

[wellsiau-aws]

Terraform Core Version

1.10.2

AWS Provider Version

5.84.0

Affected Resource(s)

In September 2024 Amazon S3 updated the default transition behavior for small objects. Full details can be found here

Summary of changes:

• New default transition behavior post Sept 2024: object smaller than 128 KB will not transition to any storage class.
• Previous default transition behavior pre Sept 2024 : object smaller than 128 KB is allowed to transitioned only to Glacier and Deep Archive
• Configurations created before September 2024 retain the previous transition behavior unless you modify them.
• If you create, edit, or delete rules, the default transition behavior for your configuration changes to the new updated behavior
• If you modify the life cycle configuration and wish to keep the previous behavior, you must explicitly set header / attribute transition_default_minimum_object_size = varies_by_storage_class

Two scenario where the current implementation could cause unintended results.

Scenario 1 - explicit attribute declaration followed by absent of attribute

Suppose that user intentionally set transition_default_minimum_object_size = varies_by_storage_class during initial apply:

resource "aws_s3_bucket_lifecycle_configuration" "bucket-config" {
  bucket = aws_s3_bucket.this.bucket
  transition_default_minimum_object_size = "varies_by_storage_class"

  rule {
    id = "transition_accesslogs_to_glacier"
    . . . 
  }
}

and then decided that they do not want to set this attribute anymore. In this case, user assumes that new behavior will take precendence when attribute transition_default_minimum_object_size was not explicitly set.

resource "aws_s3_bucket_lifecycle_configuration" "bucket-config" {
  bucket = aws_s3_bucket.this.bucket
  # transition_default_minimum_object_size = "varies_by_storage_class"
  rule {
    id = "transition_accesslogs_to_glacier"
    . . . 
  }
}

In this scenario, Terraform will ignore the changes and keep the old behavior.

aws_s3_bucket.this: Refreshing state... [id=wellsiau-202520250125014543455700000001]
aws_s3_bucket_lifecycle_configuration.bucket-config: Refreshing state... [id=wellsiau-202520250125014543455700000001]

No changes. Your infrastructure matches the configuration.

Even if user made changes to other attributes, Terraform will continue to honor the old value for transition_default_minimum_object_size = "varies_by_storage_class"

For example, if user change the transition rule from 30 to 60 days:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket_lifecycle_configuration.bucket-config will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "bucket-config" {
        id                                     = "wellsiau-202520250125014543455700000001"
        # (3 unchanged attributes hidden)

      ~ rule {
            id     = "transition_accesslogs_to_glacier"
            # (2 unchanged attributes hidden)

          - transition {
              - days          = 30 -> null
              - storage_class = "GLACIER" -> null
                # (1 unchanged attribute hidden)
            }
          + transition {
              + days          = 60
              + storage_class = "GLACIER"
                # (1 unchanged attribute hidden)
            }

            # (2 unchanged blocks hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

My concern is that this behavior is not exactly match what is documented in S3 documentation here:

Configurations created before September 2024 retain the previous transition behavior unless you modify them. That is, if you create, edit, or delete rules, the default transition behavior for your configuration changes to the updated behavior. If your use case requires, you can change the default transition behavior so that objects smaller than 128KB will transition to S3 Glacier and S3 Glacier Deep Archive. To do this, use the optional x-amz-transition-default-minimum-object-size header in a PutBucketLifecycleConfiguration request.

Scenario 2 - provider upgrade can introduce unintended old behavior

Suppose that user previously set lifecycle configuration prior to Sept 2024. Notice that transition_default_minimum_object_size does not existed at the time.

resource "aws_s3_bucket_lifecycle_configuration" "bucket-config" {
  bucket = aws_s3_bucket.this.bucket

  rule {
    id = "transition_accesslogs_to_glacier"

    expiration {
      days = 1095
    }

    transition {
      days          = 30
      storage_class = "GLACIER"
    }

    status = "Enabled"
  }
}

Then user upgraded the provider and made minor change to the configuration (modify transition day from 30 -> 60). Notice that transition_default_minimum_object_size still is not explicitly added to the configuration:

resource "aws_s3_bucket_lifecycle_configuration" "bucket-config" {
  bucket = aws_s3_bucket.this.bucket

  rule {
    id = "transition_accesslogs_to_glacier"

    expiration {
      days = 1095
    }

    transition {
      days          = 60  # user add this new changes
      storage_class = "GLACIER"
    }

    status = "Enabled"
  }
}

During terraform plan / apply, interesting sequence of API response can be observed as follow:

• First, Terraform calls S3 API to get the resource state.
• Because this lifecycle rule is created post Sept 2024, S3 API WILL return attribute transition_default_minimum_object_size = "varies_by_storage_class"
• Because this attribute was not explicitly declared in the configuration (nor the statefile at this moment), I believe that Terraform plan wont even show the value for this attribute. (I dont have the right bucket setup to replicate this, but I could somewhat replicate this issue by running terraform import).
• Terraform plan will show changes to other attribute such as transition days from 30 to 60 days.
• Terraform apply will send PUT to S3 with attribute transition_default_minimum_object_size = "varies_by_storage_class"

As the result, user might trigger transition of object smaller than 128 KB unknowingly.

Proposal

We could explicitly set default value in the schema, so that when transition_default_minimum_object_size is not specified, it will always default to all_storage_classes_128K.

References

• S3 documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html#lifecycle-configuration-constraints

• Example S3 GetBucketLifecycleConfiguration from bucket created pre Sept 2024:

aws s3api get-bucket-lifecycle-configuration --bucket "xxxxxx-redacted-xxxxxx" | jq "."
{
  "TransitionDefaultMinimumObjectSize": "varies_by_storage_class",
  "Rules": [
    {
      "Expiration": {
        "Days": 3650
      },
      "ID": "Delete CloudTrail logs that are older than 10 years",
      "Filter": {
        "Prefix": "AWSLogs/"
      },
      "Status": "Enabled"
    },
    {
      "ID": "IntelligentTier",
      "Filter": {
        "Prefix": ""
      },
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 31,
          "StorageClass": "INTELLIGENT_TIERING"
        }
      ]
    }
  ]
}

Expected Behavior

aws_s3_bucket_lifecycle_configuration should default to set attribute transition_default_minimum_object_size to all_storage_classes_128K when it's not explicitly specified.

Actual Behavior

aws_s3_bucket_lifecycle_configuration will set attribute transition_default_minimum_object_size based on the returned response from S3 API.

Relevant Error/Panic Output Snippet

Terraform Configuration Files

see description

Steps to Reproduce

see description

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.86.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.