Fix bug when installer does not create CCM secret for Nutanix workload cluster

[adiantum]

Description of changes:
Fix bug when installer does not create CCM secret for Nutanix workload cluster
Following fixes applied:

• fixed templates
• fixed reconciler
• improved tests

Testing (if applicable):
Management cluster

$ eksctl anywhere create cluster -f ./mgmt-cluster.yaml -v10 --bundles-override bin/local-bundle-release.yaml
2024-05-21T16:47:16.491Z        V6      Executing command       {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2024-05-21T16:47:16.513Z        V6      Executing command       {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2024-05-21T16:47:16.564Z        V4      Reading bundles manifest        {"url": "bin/local-bundle-release.yaml"}
2024-05-21T16:47:16.584Z        V4      Using CAPI provider versions    {"Core Cluster API": "v1.6.0+ae39aac", "Kubeadm Bootstrap": "v1.6.0+49ef750", "Kubeadm Control Plane": "v1.6.0+a7122b7", "External etcd Bootstrap": "v1.0.10+1ceb898", "External etcd Controller": "v1.0.17+5e33062", "Cluster API Provider Nutanix": "v1.3.4"}
2024-05-21T16:47:16.671Z        V5      Retrier:        {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-05-21T16:47:16.671Z        V2      Pulling docker image    {"image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T16:47:16.671Z        V6      Executing command       {"cmd": "/usr/bin/docker pull public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T16:47:17.247Z        V5      Retry execution successful      {"retries": 1, "duration": "575.366823ms"}
2024-05-21T16:47:17.247Z        V3      Initializing long running container     {"name": "eksa_1716310036671798518", "image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T16:47:17.247Z        V6      Executing command       {"cmd": "/usr/bin/docker run -d --name eksa_1716310036671798518 --network host -w /home/ubuntu/mgmt016 -v /var/run/docker.sock:/var/run/docker.sock -v /home/ubuntu/mgmt016:/home/ubuntu/mgmt016 -v /home/ubuntu/mgmt016:/home/ubuntu/mgmt016 --entrypoint sleep public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321 infinity"}
2024-05-21T16:47:17.395Z        V0      Using the new workflow using the controller for management cluster create
2024-05-21T16:47:17.395Z        V4      Task start      {"task_name": "setup-validate"}
2024-05-21T16:47:17.395Z        V0      Performing setup and validations
2024-05-21T16:47:17.395Z        V0      ValidateClusterSpec for Nutanix datacenter      {"NutanixDatacenter": "mgmt"}
2024-05-21T16:47:20.993Z        V0      ✅ Nutanix Provider setup is valid
2024-05-21T16:47:20.993Z        V0      ✅ Validate OS is compatible with registry mirror configuration
2024-05-21T16:47:20.993Z        V0      ✅ Validate certificate for registry mirror
2024-05-21T16:47:20.994Z        V0      ✅ Validate authentication for git provider
2024-05-21T16:47:20.994Z        V0      ✅ Validate cluster's eksaVersion matches EKS-A version
2024-05-21T16:47:20.994Z        V4      Task finished   {"task_name": "setup-validate", "duration": "3.598307666s"}
...
2024-05-21T16:56:14.066Z        V5      Retry execution successful      {"retries": 1, "duration": "169.568405ms"}
2024-05-21T16:56:14.066Z        V5      Retrier:        {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-05-21T16:56:14.066Z        V4      Deleting kind cluster   {"name": "mgmt-eks-a-cluster"}
2024-05-21T16:56:14.066Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310036671798518 kind delete cluster --name mgmt-eks-a-cluster"}
2024-05-21T16:56:15.400Z        V5      Retry execution successful      {"retries": 1, "duration": "1.333492054s"}
2024-05-21T16:56:15.400Z        V0      🎉 Cluster created!
2024-05-21T16:56:15.400Z        V4      Task finished   {"task_name": "delete-kind-cluster", "duration": "1.795673206s"}
2024-05-21T16:56:15.400Z        V4      ----------------------------------
2024-05-21T16:56:15.400Z        V4      Task start      {"task_name": "install-curated-packages"}
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
2024-05-21T16:56:15.400Z        V0      Enabling curated packages on the cluster
...

Workload cluster

2024-05-21T17:01:57.443Z        V6      Executing command       {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2024-05-21T17:01:57.466Z        V6      Executing command       {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2024-05-21T17:01:57.517Z        V4      Reading bundles manifest        {"url": "bin/local-bundle-release.yaml"}
2024-05-21T17:01:57.537Z        V4      Using CAPI provider versions    {"Core Cluster API": "v1.6.0+ae39aac", "Kubeadm Bootstrap": "v1.6.0+49ef750", "Kubeadm Control Plane": "v1.6.0+a7122b7", "External etcd Bootstrap": "v1.0.10+1ceb898", "External etcd Controller": "v1.0.17+5e33062", "Cluster API Provider Nutanix": "v1.3.4"}
2024-05-21T17:01:57.628Z        V5      Retrier:        {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-05-21T17:01:57.628Z        V2      Pulling docker image    {"image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T17:01:57.628Z        V6      Executing command       {"cmd": "/usr/bin/docker pull public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T17:01:58.198Z        V5      Retry execution successful      {"retries": 1, "duration": "570.833773ms"}
2024-05-21T17:01:58.199Z        V3      Initializing long running container     {"name": "eksa_1716310917628126682", "image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321"}
2024-05-21T17:01:58.199Z        V6      Executing command       {"cmd": "/usr/bin/docker run -d --name eksa_1716310917628126682 --network host -w /home/ubuntu/mgmt016 -v /var/run/docker.sock:/var/run/docker.sock -v /home/ubuntu/mgmt016/mgmt:/home/ubuntu/mgmt016/mgmt -v /home/ubuntu/mgmt016:/home/ubuntu/mgmt016 -v /home/ubuntu/mgmt016:/home/ubuntu/mgmt016 --entrypoint sleep public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.18.6-eks-a-v0.0.0-dev-build.8321 infinity"}
2024-05-21T17:01:58.354Z        V4      Task start      {"task_name": "setup-validate-create"}
2024-05-21T17:01:58.354Z        V0      ValidateClusterSpec for Nutanix datacenter      {"NutanixDatacenter": "mgmt-wrk1"}
2024-05-21T17:02:04.544Z        V0      ✅ Workload cluster's nutanix Provider setup is valid
2024-05-21T17:02:04.544Z        V0      ✅ Validate OS is compatible with registry mirror configuration
2024-05-21T17:02:04.544Z        V0      ✅ Validate certificate for registry mirror
2024-05-21T17:02:04.544Z        V0      ✅ Validate authentication for git provider
2024-05-21T17:02:04.544Z        V0      ✅ Validate cluster's eksaVersion matches EKS-A version
2024-05-21T17:02:04.545Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get clusters.cluster.x-k8s.io -o json --kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig --namespace eksa-system"}
2024-05-21T17:02:04.737Z        V0      ✅ Validate cluster name
2024-05-21T17:02:04.737Z        V0      ✅ Validate gitops
2024-05-21T17:02:04.737Z        V5      skipping ValidateIdentityProviderNameIsUnique
2024-05-21T17:02:04.737Z        V0      ✅ Validate identity providers' name
2024-05-21T17:02:04.737Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get customresourcedefinition clusters.cluster.x-k8s.io --kubeconfig mgmt
/mgmt-eks-a-cluster.kubeconfig"}
2024-05-21T17:02:04.881Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get customresourcedefinition clusters.anywhere.eks.amazonaws.com --kubec
onfig mgmt/mgmt-eks-a-cluster.kubeconfig"}
2024-05-21T17:02:05.028Z        V0      ✅ Validate management cluster has eksa crds
2024-05-21T17:02:05.028Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get clusters.anywhere.eks.amazonaws.com -A -o jsonpath={.items[0]} --kub
econfig mgmt/mgmt-eks-a-cluster.kubeconfig --field-selector=metadata.name=mgmt"}
2024-05-21T17:02:05.206Z        V0      ✅ Validate management cluster name is valid
2024-05-21T17:02:05.206Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get clusters.anywhere.eks.amazonaws.com -A -o jsonpath={.items[0]} --kub
econfig mgmt/mgmt-eks-a-cluster.kubeconfig --field-selector=metadata.name=mgmt"}
2024-05-21T17:02:05.385Z        V0      ✅ Validate management cluster eksaVersion compatibility
2024-05-21T17:02:05.385Z        V6      Executing command       {"cmd": "/usr/bin/docker exec -i eksa_1716310917628126682 kubectl get --ignore-not-found -o json --kubeconfig mgmt/mgmt-eks-a-cluster.kube
config EKSARelease.v1alpha1.anywhere.eks.amazonaws.com --namespace eksa-system eksa-v0-0-0-dev"}
2024-05-21T17:02:05.530Z        V0      ✅ Validate eksa release components exist on management cluster
2024-05-21T17:02:05.530Z        V4      Task finished   {"task_name": "setup-validate-create", "duration": "7.175213484s"}
2024-05-21T17:02:05.530Z        V4      ----------------------------------
2024-05-21T17:02:05.530Z        V4      Task start      {"task_name": "create-workload-cluster"}
2024-05-21T17:02:05.530Z        V0      Creating workload cluster
...
2024-05-21T17:23:10.516Z        V4      Task finished   {"task_name": "create-workload-cluster", "duration": "21m4.98645452s"}
2024-05-21T17:23:10.516Z        V4      ----------------------------------
2024-05-21T17:23:10.516Z        V4      Task start      {"task_name": "gitops-manager-install"}
2024-05-21T17:23:10.516Z        V0      Installing GitOps Toolkit on workload cluster
2024-05-21T17:23:10.516Z        V0      GitOps field not specified, bootstrap flux skipped
2024-05-21T17:23:10.516Z        V4      Task finished   {"task_name": "gitops-manager-install", "duration": "24.769µs"}
2024-05-21T17:23:10.516Z        V4      ----------------------------------
2024-05-21T17:23:10.516Z        V4      Task start      {"task_name": "write-cluster-config"}
2024-05-21T17:23:10.516Z        V0      Writing cluster config file
2024-05-21T17:23:10.519Z        V0      🎉 Cluster created!
2024-05-21T17:23:10.519Z        V4      Task finished   {"task_name": "write-cluster-config", "duration": "2.649362ms"}
2024-05-21T17:23:10.519Z        V4      ----------------------------------
2024-05-21T17:23:10.519Z        V4      Tasks completed {"duration": "21m12.164496416s"}
2024-05-21T17:23:10.519Z        V3      Cleaning up long running container      {"name": "eksa_1716310917628126682"}
2024-05-21T17:23:10.519Z        V6      Executing command       {"cmd": "/usr/bin/docker rm -f -v eksa_1716310917628126682"}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[eks-distro-bot]

Hi @adiantum. Thanks for your PR.

I'm waiting for a aws member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abhinavmpandey08]

/ok-to-test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.40%. Comparing base (f631874) to head (3a5206d).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8191   +/-   ##
=======================================
  Coverage   73.40%   73.40%           
=======================================
  Files         577      577           
  Lines       35900    35908    +8     
=======================================
+ Hits        26352    26360    +8     
  Misses       7882     7882           
  Partials     1666     1666           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[abhinavmpandey08]

/approve

[eks-distro-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavmpandey08

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavmpandey08]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[adiantum]

/cherry-pick release-0.19

[eks-distro-pr-bot]

@adiantum: only aws org members may request cherry picks. You can still do the cherry-pick manually.

In response to this:

/cherry-pick release-0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.