Unable to modify [settings.kubernetes] with bootstrap_extra_args

[travisnburton]

Description

Unable to pass values for [settings.kubernetes] via bootstrap_extra_args because EKS is already managing [settings.kubernetes]

duplicate [settings.kubernetes] key provided via bootstrap_extra_args:

    bootstrap_extra_args = <<-EOT
      # extra args added
      [settings.kubernetes]
      container-log-max-size = "20Mi"
      container-log-max-files = "2"
      EOT

Versions

• Terraform:
  Terraform v1.1.7 on darwin_arm64
• Provider(s):
  provider registry.terraform.io/hashicorp/aws v4.6.0
  provider registry.terraform.io/hashicorp/cloudinit v2.2.0
  provider registry.terraform.io/hashicorp/helm v2.4.1
  provider registry.terraform.io/hashicorp/null v3.1.1
  provider registry.terraform.io/hashicorp/tls v3.1.0
• Module:
  18.14.1, 18.7.2

Reproduction

With BottlerocketOS + eks_managed_node_group - use bootstrap_extra_args to add values for settings.kubernetes.container-log-max-size and settings.kubernetes.container-log-max-files

Code Snippet to Reproduce

  eks_managed_node_group_defaults = {
    description = "EKS managed node group launch template"
    timeouts = {
      create = "20m"
      delete = "20m"
    }

    ami_type = "BOTTLEROCKET_x86_64"
    platform = "bottlerocket" 

    bootstrap_extra_args = <<-EOT
      # extra args added
      [settings.kubernetes]
      container-log-max-size = "20Mi"
      container-log-max-files = "2"
      EOT

    instance_types = ["t3.medium"] 
    capacity_type  = "ON_DEMAND"
    ebs_optimized  = true
    enable_monitoring = true

    block_device_mappings = {
      xvdb = { 
        device_name = "/dev/xvdb"
        ebs = {
          volume_size           = 75
          volume_type           = "gp3"
          iops                  = 3000
          throughput            = 150
          encrypted             = true
          kms_key_id            = aws_kms_key.ebs.arn
          delete_on_termination = true
        }
      }
    }

Expected behavior

Expect values to be set as defined

Actual behavior

Nodes are unable to join managed node group because [settings.kubernetes] is invalidated by duplicate entry

Additional context

see TOML - Allow duplicate keys #697

[bryantbiggs]

unfortunately, we do not have any control over this at the module level. the EKS managed service injects its own user data to bootstrap nodes (for both the Linux 2 and Bottlerocket based OS's)

I would suggest either asking the question upstream https://github.com/bottlerocket-os/bottlerocket or filing a ticket with AWS support. For reference, we face similar issues with the Linux 2 based nodes awslabs/amazon-eks-ami#844 but users have some flexibility to work around with some bash script fu

[travisnburton]

upstream issue raised - bottlerocket-os/bottlerocket#2024

[travisnburton]

Confirmed with EKS MNG team that TOML merging is supported - bottlerocket-os/bottlerocket#2024 (comment)

My issue was an incorrect type (string instead of int) for settings.kubernetes.container-log-max-files

The config below works as expected:

bootstrap_extra_args = <<-EOT
      # extra args added
      [settings.kubernetes]
      container-log-max-size = "20Mi"
      container-log-max-files = 2
      EOT

[bryantbiggs]

awesome, glad you were able to get it sorted 🎉

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.