[Bug]: aws_dms_replication_task is requiring a cdcPath parameter when one has been provided on the aws_dms_s3_endpoint for the task

[rossbush]

Terraform Core Version

1.3.6

AWS Provider Version

4.46.0

Affected Resource(s)

aws_dms_replication_task

Expected Behavior

During the terraform apply the cdcpath should be picked up from the source_endpoint_arn which is a aws_dms_s3_endpoint with cdc_path set.

Actual Behavior

During the terraform apply an invalid parameter error is being returned for all aws_dms_replication_task configured with a s3 source like above.

Relevant Error/Panic Output Snippet

2022-12-09T17:00:07.5370387Z Error: error creating DMS Replication Task (s3-rds-sqlserver-replication-task-crem-coalesce-sbx-vp-retaildemo-v05): InvalidParameterValueException: Parameter cdcPath is required when task is configured for ongoing replication with Amazon S3 as a source.
2022-12-09T17:00:07.5371577Z 	status code: 400, request id: 71ec429e-71c0-4beb-9594-f182d12cf2c5
2022-12-09T17:00:07.5371848Z 
2022-12-09T17:00:07.5372232Z   with aws_dms_replication_task.s3_to_rds_sqlserver_replication_task["VP_RETAILDEMO_V05"],
2022-12-09T17:00:07.5373041Z   on dms-bronze.tf line 249, in resource "aws_dms_replication_task" "s3_to_rds_sqlserver_replication_task":
2022-12-09T17:00:07.5373601Z  249:  resource "aws_dms_replication_task" "s3_to_rds_sqlserver_replication_task" {

Terraform Configuration Files

#Using the new aws_dms_s3_endpoint resource below in dms, the cdc_path has been specified:

resource "aws_dms_s3_endpoint" "silver_source_endpoint" {
  for_each = var.database_names

  bucket_name                 = module.s3_silver.s3_bucket_name
  bucket_folder               = each.value
  csv_delimiter               = null 
  cdc_path                    = "cdc-files"
  csv_row_delimiter           = null  
  date_partition_enabled      = false
  endpoint_id                 = "silver-source-endpoint-${local.name}-${lower(replace(each.value, "_", "-"))}"
  endpoint_type               = "source"  
  external_table_definition   = local.table_definition
  ignore_header_rows          = 1    
  rfc_4180                    = true
  service_access_role_arn     = aws_iam_role.dms_s3_role.arn
  ssl_mode                    = "none"    

  tags = merge(local.common_tags, tomap({ "Name" = "${local.name}-${each.value}" }))

  #depends_on = [null_resource.sleep]
    depends_on = [
    aws_iam_role_policy_attachment.dms-vpc-role-AmazonDMSVPCManagementRole
  ]

  lifecycle {
   ignore_changes = []
 }
}

#When creating an aws_dms_replication task, the resource above is used as the source_endpoint_arn:

 resource "aws_dms_replication_task" "s3_to_rds_sqlserver_replication_task" {
   for_each = var.database_names

   migration_type           = "full-load-and-cdc"
   replication_instance_arn = aws_dms_replication_instance.dms_instance_silver.replication_instance_arn
   replication_task_id      = format("s3-rds-sqlserver-replication-task-%s-%s", local.name, lower(replace(each.value, "_", "-")))

   source_endpoint_arn = aws_dms_s3_endpoint.silver_source_endpoint[each.value].endpoint_arn
   target_endpoint_arn = aws_dms_endpoint.silver_target_sqlserver_endpoint[each.value].endpoint_arn

   replication_task_settings = local.silver_task_settings
   table_mappings            = local.silver_task_table_mappings

   tags = local.common_tags
 }

Steps to Reproduce

1. Create an aws_dms_s3_endpoint and set the cdc_path to a valid bucket.
2. Create an aws_dms_replication_task and use the resource created in step 1 as the task's source

Debug Output

No response

Panic Output

No response

Important Factoids

I am not sure if the cdc-path needs to be specified in the aws_dms_replication_task.replication_task_settings. If so, I did not see in any documentation where this value should be set.

References

#13491

#13491

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[rossbush]

It appears that there is a CDC Path property on the updated aws dms endpoint but the aws_dms_s3_endpoint.cdc_path value is ending up on the aws dms endpoint's EndpointSettings.CdcPath.

[YakDriver]

@rossbush Thanks for reporting this! This is odd. I'll take a closer look. I feel like something strange is happening on the AWS side but it's also possible I missed something.

[rossbush]

@rossbush Thanks for reporting this! This is odd. I'll take a closer look. I feel like something strange is happening on the AWS side but it's also possible I missed something.

@YakDriver - Thanks! This seems to have been confirmed by AWS as an issue (see link below). The ?new? DMS task's CDC Path property, (not the one in the Endpoint Setting or the soon to be depreciated Extra Connection Info) cannot be set via the command line; however, if the DMS Endpoint is a Source endpoint and of type S3 bucket, there must be a value in this field. Sadly, we do not have edit permissions to our endpoints above dev. We are stuck :/

https://repost.aws/questions/QUJu5_5Z05TuSE3VBwTv-xRQ/how-to-set-the-cdc-path-of-a-dms-endpoint-of-type-s-3

[github-actions]

This functionality has been released in v4.50.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.