Terraform Crash v0.12.2

[hdryx]

Hi,

Terraform crashed for the first time for my project of an ECS cluster. I'm using the v0.12.2

This is the last part of the crash.log :

------
2019/06/26 12:31:48 [TRACE] Executing graph transform *terraform.OrphanResourceCountTransformer
2019/06/26 12:31:48 [TRACE] Completed graph transform *terraform.OrphanResourceCountTransformer (no changes)
2019/06/26 12:31:48 [TRACE] Executing graph transform *terraform.AttachStateTransformer
2019/06/26 12:31:48 [DEBUG] Resource instance state not found for node "aws_ecs_service.service[0]", instance aws_ecs_service.service[0]
2019/06/26 12:31:48 [TRACE] Completed graph transform *terraform.AttachStateTransformer (no changes)
2019/06/26 12:31:48 [TRACE] Executing graph transform *terraform.TargetsTransformer
2019/06/26 12:31:48 [TRACE] Completed graph transform *terraform.TargetsTransformer (no changes)
2019/06/26 12:31:48 [TRACE] Executing graph transform *terraform.ReferenceTransformer
2019/06/26 12:31:48 [DEBUG] ReferenceTransformer: "aws_ecs_service.service[0]" references: []
2019/06/26 12:31:48 [TRACE] Completed graph transform *terraform.ReferenceTransformer (no changes)
2019/06/26 12:31:48 [TRACE] Executing graph transform *terraform.RootTransformer
2019/06/26 12:31:48 [TRACE] Completed graph transform *terraform.RootTransformer (no changes)
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service": entering dynamic subgraph
2019/06/26 12:31:48 [TRACE] dag/walk: updating graph
2019/06/26 12:31:48 [TRACE] dag/walk: added new vertex: "aws_ecs_service.service[0]"
2019/06/26 12:31:48 [TRACE] dag/walk: visiting "aws_ecs_service.service[0]"
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service[0]": starting visit (*terraform.NodeRefreshableManagedResourceInstance)
2019/06/26 12:31:48 [TRACE] NodeRefreshableManagedResourceInstance: aws_ecs_service.service[0] has no existing state to refresh
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service[0]": evaluating
2019/06/26 12:31:48 [TRACE] [walkRefresh] Entering eval tree: aws_ecs_service.service[0]
2019/06/26 12:31:48 [TRACE] <root>: eval: *terraform.EvalSequence
2019/06/26 12:31:48 [TRACE] <root>: eval: *terraform.EvalGetProvider
2019/06/26 12:31:48 [TRACE] <root>: eval: *terraform.EvalReadState
2019/06/26 12:31:48 [TRACE] EvalReadState: reading state for aws_ecs_service.service[0]
2019/06/26 12:31:48 [TRACE] EvalReadState: no state present for aws_ecs_service.service[0]
2019/06/26 12:31:48 [TRACE] <root>: eval: *terraform.EvalDiff
2019/06/26 12:31:48 [TRACE] GetResourceInstance: aws_ecs_task_definition.task_definition has multiple keyed instances
2019/06/26 12:31:48 [TRACE] GetResourceInstance: aws_ecs_cluster.cluster is a single instance
2019/06/26 12:31:48 [TRACE] GetResourceInstance: aws_alb_target_group.alb_target_group has multiple keyed instances
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service[0]": visit complete
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service": dynamic subgraph completed successfully
2019/06/26 12:31:48 [TRACE] vertex "aws_ecs_service.service": visit complete
2019/06/26 12:31:48 [TRACE] dag/walk: visiting "provider.aws (close)"
2019/06/26 12:31:48 [TRACE] vertex "provider.aws (close)": starting visit (*terraform.graphNodeCloseProvider)
2019/06/26 12:31:48 [TRACE] vertex "provider.aws (close)": evaluating
2019/06/26 12:31:48 [TRACE] [walkRefresh] Entering eval tree: provider.aws (close)
2019/06/26 12:31:48 [TRACE] <root>: eval: *terraform.EvalCloseProvider
2019/06/26 12:31:48 [TRACE] GRPCProvider: Close
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xed2aa3]

goroutine 304 [running]:
github.com/hashicorp/terraform/terraform.(*evaluationStateData).getResourceInstancesAll(0xc000c7f830, 0x4d, 0xc0004bf0e0, 0x14, 0xc000380e70, 0x10, 0xc000454c09, 0x7, 0x4c, 0x23, ...)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/evaluate.go:704 +0x11a3
github.com/hashicorp/terraform/terraform.(*evaluationStateData).GetResourceInstance(0xc000c7f830, 0x4d, 0xc0004bf0e0, 0x14, 0xc000380e70, 0x10, 0x0, 0x0, 0xc000454c09, 0x7, ...)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/evaluate.go:593 +0x1360
github.com/hashicorp/terraform/lang.(*Scope).evalContext(0xc0005d7ea0, 0xc000115d40, 0x17, 0x17, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40c0d8)
	/opt/gopath/src/github.com/hashicorp/terraform/lang/eval.go:253 +0xfcc
github.com/hashicorp/terraform/lang.(*Scope).EvalContext(...)
	/opt/gopath/src/github.com/hashicorp/terraform/lang/eval.go:160
github.com/hashicorp/terraform/lang.(*Scope).EvalBlock(0xc0005d7ea0, 0x21e97e0, 0xc000beaae0, 0xc000218a50, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc0007b0000, ...)
	/opt/gopath/src/github.com/hashicorp/terraform/lang/eval.go:52 +0xf1
github.com/hashicorp/terraform/terraform.(*BuiltinEvalContext).EvaluateBlock(0xc000375450, 0x21e97e0, 0xc000beaae0, 0xc000218a50, 0x0, 0x0, 0x21eabe0, 0xc00003e468, 0x1d5fc80, 0xc000bdb080, ...)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_context_builtin.go:277 +0x2b6
github.com/hashicorp/terraform/terraform.(*EvalDiff).Eval(0xc000115800, 0x221c700, 0xc000375450, 0x2, 0x2, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_diff.go:138 +0x1e1
github.com/hashicorp/terraform/terraform.EvalRaw(0x21a9320, 0xc000115800, 0x221c700, 0xc000375450, 0x0, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:57 +0x131
github.com/hashicorp/terraform/terraform.(*EvalSequence).Eval(0xc000327920, 0x221c700, 0xc000375450, 0x2, 0x2, 0xebec75, 0x21a9720)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_sequence.go:20 +0xfd
github.com/hashicorp/terraform/terraform.EvalRaw(0x21a9640, 0xc000327920, 0x221c700, 0xc000375450, 0x1a649a0, 0x3196104, 0x19d1d20, 0xc001620460)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:57 +0x131
github.com/hashicorp/terraform/terraform.Eval(0x21a9640, 0xc000327920, 0x221c700, 0xc000375450, 0xc000327920, 0x21a9640, 0xc000327920, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:35 +0x4d
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x1cee980, 0xc00000fa38, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/graph.go:90 +0xf40
github.com/hashicorp/terraform/dag.(*Walker).walkVertex(0xc0009be400, 0x1cee980, 0xc00000fa38, 0xc00148a440)
	/opt/gopath/src/github.com/hashicorp/terraform/dag/walk.go:392 +0x353
created by github.com/hashicorp/terraform/dag.(*Walker).Update
	/opt/gopath/src/github.com/hashicorp/terraform/dag/walk.go:314 +0xa9b

Thanks for your help.

[hposca]

I just had a similar problem, which I was able to solve.

The main hint was that on the crash.log a line like this one appeared: 2019/06/28 15:06:01 [TRACE] GetResourceInstance: aws_instance.test has multiple keyed instances

I was doing some tests with the remote-exec provisioner on some aws_instance, but the remote execution, never started due to some connectivity issues and terraform error-ed as expected. BUT it never killed the instance.
Then I tried again, and it failed again.
(Up to this point no terraform panic)

As in the next execution terraform created a new instance (without destroying the previously created one) I manually deleted the instance that was previously created.
(Up to this point no terraform panic)

Then, IIRC, I deleted a instance before executing terraform apply.
(Now, I had a terraform panic)

After some digging into the terraform state file I realized that terraform still had some pointers to two instances that I deleted previously (at this point I had no more instances running on AWS). But on the output section I had a pointer to instance A and on the resources part I had pointer to instance B and A. Maybe this is what cause the aws_instance.test has multiple keyed instances message on the crash log.

After deleting the references to instance B things went back to normal and no more terraform panic :)

My terraform version was 0.12.2 and I even tried 0.12.3 (after the terraform panic) without success.

Hope that this at least help someone with a similar issue.

[bpetermannS11]

I created a pull request that should fix this issue.
If there is a multi-instance resource (count > 1) where one of the instances was deleted from the deployment, but is still present in the terraform state, terraform plan crashed.

[jbardin]

closed by #21979

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.