panic: interface conversion: interface {} is nil, not map[string]interface

[madengr00]

---

name: 🐛 Bug Report
about: Error occurring (panic: interface conversion: interface {} is nil, not map[string]interface) when applying a plan in TFE

---

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• Terraform v 1.1.5
• provider.aws v4.14.0

Affected Resource(s)

replaced name with EXAMPLE for privacy

• aws_msk_cluster

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      version = "~> 4.0"
    }
  }
  required_version = "~>1.1.5"
}


resource "aws_msk_cluster" "<EXAMPLE>-msk-cluster" {
  broker_node_group_info {
    az_distribution = "DEFAULT"
    client_subnets  = var.vpc_subnets[var.envr]
    ebs_volume_size = "1000"
    instance_type   = "kafka.m5.large"
    security_groups = var.vpc_security_groups[var.envr]
  }

  client_authentication {
    sasl {
      iam   = "true"
      scram = "true"
    }
    tls {
      certificate_authority_arns = null
    }
  }

  cluster_name = "<EXAMPLE>-msk-${var.msk_cluster_env_name[var.envr]}-001"

  encryption_info {
     encryption_at_rest_kms_key_arn = data.aws_kms_key.aws_kms_kafka.arn

    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = "true"
    }
  }

  enhanced_monitoring = var.msk_cluster_enhanced_monitoring[var.envr]
  kafka_version       = "2.6.0"

  logging_info {
    broker_logs {
      cloudwatch_logs {
        enabled   = "true"
        log_group = var.msk_cluster_log_name[var.envr]
      }

      firehose {
        enabled = "false"
      }

      s3 {
        enabled = "false"
      }
    }
  }

  number_of_broker_nodes = "3"

  open_monitoring {
    prometheus {
      jmx_exporter {
        enabled_in_broker = "true"
      }

      node_exporter {
        enabled_in_broker = "true"
      }
    }
  }

  tags = {
    "ConfigBy" = "TF: <EXAMPLE>-kafka-${var.envr}"
    "Expires"  = "2023-05-02"
    "Name"     = "<EXAMPLE>-msk-${var.msk_cluster_env_name[var.envr]}-001"
    "SME"      = "teamname"
  }
}

Debug Output

Panic Output

Stack trace from the terraform-provider-aws_v4.14.0_x5 plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 113 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/kafka.expandClientAuthentication(0xc0022c3e00)
github.com/hashicorp/terraform-provider-aws/internal/service/kafka/cluster.go:918 +0x374
github.com/hashicorp/terraform-provider-aws/internal/service/kafka.resourceClusterUpdate({0xa4b7c20, 0xc002138a80}, 0xc002166500, {0x7f68540, 0xc000a86000})
github.com/hashicorp/terraform-provider-aws/internal/service/kafka/cluster.go:773 +0x36d7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xa4b7c20, {0xa4b7c20, 0xc002138a80}, 0xd, {0x7f68540, 0xc000a86000})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000ec02a0, {0xa4b7c20, 0xc002138a80}, 0xc00214add0, 0xc002166380, {0x7f68540, 0xc000a86000})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:847 +0x9e5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00000cae0, {0xa4b7b78, 0xc001ec9640}, 0xc001dc3d60)
github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/grpc_provider.go:1021 +0xe3c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0018f8f00, {0xa4b7c20, 0xc002138480}, 0xc000fc79d0)
github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/tf5server/server.go:812 +0x56b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x91626e0, 0xc0018f8f00}, {0xa4b7c20, 0xc002138480}, 0xc001c0b380, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fcc40, {0xa5c3290, 0xc000c121a0}, 0xc0021327e0, 0xc0018f3860, 0x101abd00, 0x0)
google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc0001fcc40, {0xa5c3290, 0xc000c121a0}, 0xc0021327e0, 0x0)
google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.45.0/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.45.0/server.go:919 +0x294

Error: The terraform-provider-aws_v4.14.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Expected Behavior

Plan looks clean in terraform enterprise.
On apply in Terraform Enterprise, we should have been apply two changes;

1. Update in place of cluster changing the client authentication-->sasl--> iam = "true"

Actual Behavior

panic error on apply

Steps to Reproduce

Our apply process occurs in a Terraform Enterprise workspace.

1. terraform apply

Important Factoids

Cluster was originally configured through the UI on AWS.
Created the terraform retroactively and imported it to state. Now, trying to update the configuration with tf.
Errors occured in the plan until we updated to aws.provider 4.13.0.
Plans clean now.
We get the same apply error for both 4.13.0 and 4.14.0

References

This was the issue report that helped me figure out that I needed to use 4.13.0.
#20956 (comment)

[madengr00]

Error occurs on an apply.
"The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call."

Resources that I am trying to update and delete/create are aws_msk_cluster and aws_mskconnect

[justinretzolk]

Hey @madengr00 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to look into this, can you update the issue description to include all of the information requested in the bug report template?

[madengr00]

Hey @madengr00 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to look into this, can you update the issue description to include all of the information requested in the bug report template?

Updated as much as I could.

[lliknart]

I face the exact same issue when trying an apply (plan is working) with the following version
tf 1.1.5 / 1.1.8 / 1.2.0
aws provider: 4.11.0 / 4.14.0 / 4.15.0

debug log:
https://gist.github.com/lliknart/e142066bd3c26d504c7a9d6e4725150e

The code i wanted to deploy:
The custom module I want to use
https://gist.github.com/lliknart/d4e3d4662a0d8a542245f5999bab42d2

How the module is called:
https://gist.github.com/lliknart/b6545d3c1b4dc4276db7590657dbc056

N.B: it works when I use the old way to set lifecycle and encryption (using block in aws_s3_bucket instead of using dedicated resources)

[madengr00]

So, the tls {} block is supposed to be optional. If I remember correctly,in v 4.13.0, I had to get the plan to run clean, I had to add it in with the tls {
certificate_authority_arns = null
} to avoid plan errors.

I will test removing altogether and trying a new apply in v4.14.0.

Update: Does not work to remove it. At that point it plan to change something that isn't changing and results in a new error.
Message_: "The request does not include any updates to the security setting of the cluster. Verify the request, then try again." }

[madengr00]

Note: Updated to using v4.15.0. Still the same error.

[ewbankkit]

v4.14.0

terraform-provider-aws/internal/service/kafka/cluster.go

Lines 906 to 928 in 2bae1fb

	func expandClientAuthentication(tfMap map[string]interface{}) *kafka.ClientAuthentication {
	if tfMap == nil {
	return nil
	}
	apiObject := &kafka.ClientAuthentication{}
	if v, ok := tfMap["sasl"].([]interface{}); ok && len(v) > 0 {
	apiObject.Sasl = expandSasl(v[0].(map[string]interface{}))
	}
	if v, ok := tfMap["tls"].([]interface{}); ok && len(v) > 0 {
	apiObject.Tls = expandTls(v[0].(map[string]interface{}))
	}
	if v, ok := tfMap["unauthenticated"].(bool); ok {
	apiObject.Unauthenticated = &kafka.Unauthenticated{
	Enabled: aws.Bool(v),
	}
	}
	return apiObject
	}

v4.12.0

terraform-provider-aws/internal/service/kafka/cluster.go

Lines 800 to 812 in f932b7a

	func expandClusterTls(l []interface{}) *kafka.Tls {
	if len(l) == 0 || l[0] == nil {
	return nil
	}
	m := l[0].(map[string]interface{})
	tls := &kafka.Tls{
	CertificateAuthorityArnList: flex.ExpandStringSet(m["certificate_authority_arns"].(*schema.Set)),
	}
	return tls
	}

There is a nil check.

[ewbankkit]

I can reproduce this with a test configuration very similar to the one provided:

╷
│ Error: Plugin did not respond
│ 
│   with aws_msk_cluster.test,
│   on main.tf line 85, in resource "aws_msk_cluster" "test":
│   85: resource "aws_msk_cluster" "test" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more
│ details.
╵

Stack trace from the terraform-provider-aws plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 95 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/kafka.expandClientAuthentication(0xc0023823c0)
	/Users/ewbankkit/src/github.com/terraform-providers/terraform-provider-aws/internal/service/kafka/cluster.go:1093 +0x374
github.com/hashicorp/terraform-provider-aws/internal/service/kafka.resourceClusterUpdate({0xb1782e0, 0xc002208e10}, 0xc00220f380, {0x8c02340, 0xc000297500})
	/Users/ewbankkit/src/github.com/terraform-providers/terraform-provider-aws/internal/service/kafka/cluster.go:889 +0x3c77
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xb1782e0, {0xb1782e0, 0xc002208e10}, 0xd, {0x8c02340, 0xc000297500})
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000f74540, {0xb1782e0, 0xc002208e10}, 0xc001aa1ba0, 0xc00220ee00, {0x8c02340, 0xc000297500})
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/resource.go:847 +0x9e5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0001a5a40, {0xb178238, 0xc00206c980}, 0xc002009ef0)
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.16.0/helper/schema/grpc_provider.go:1021 +0xe3c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc001a39b80, {0xb1782e0, 0xc002208630}, 0xc001c11030)
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/tf5server/server.go:812 +0x56b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x9e0a4c0, 0xc001a39b80}, {0xb1782e0, 0xc002208630}, 0xc00202c8a0, 0x0)
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.9.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002a6700, {0xb284bb0, 0xc0009bf6c0}, 0xc002203560, 0xc001a4d5c0, 0x10ef1040, 0x0)
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc0002a6700, {0xb284bb0, 0xc0009bf6c0}, 0xc002203560, 0x0)
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x294

Error: The terraform-provider-aws plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[github-actions]

This functionality has been released in v4.16.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.