Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chef isn't installed into the container when using podman and kitchen-dokken #255

Closed
UncleOwen opened this issue Dec 1, 2021 · 7 comments · Fixed by #265
Closed

chef isn't installed into the container when using podman and kitchen-dokken #255

UncleOwen opened this issue Dec 1, 2021 · 7 comments · Fixed by #265

Comments

@UncleOwen
Copy link

🗣️ Foreword

👻 Brief Description

chef isn't installed into the container when using podman and kitchen-dokken

Version

Chef Infra Client: 17.7.29
Test Kitchen: 3.1.1
kitchen-dokken: 2.16.0

Environment

Ubuntu 20.04

Scenario

I'm trying to replace docker with podman.

Steps to Reproduce

run kitchen test in a basically empty cookbook.

kitchen.yml:

---
driver:
  name: dokken
  chef_version: latest

transport:
  name: dokken

provisioner:
  name: dokken

verifier:
  name: inspec

platforms:
- name: ubuntu-20.04
  driver:
    image: dokken/ubuntu-20.04

suites:
  - name: default
    run_list:
    - recipe[dummy]

metadata.rb:

name 'dummy'

recipes/default.rb and test/integration/default/default_test.rb are empty

Expected Result

$ DOCKER_HOST=unix:///run/user/1000/docker.sock kitchen test
-----> Starting Test Kitchen (v3.1.1)
-----> Cleaning up any prior instances of <default-ubuntu-2004>
-----> Destroying <default-ubuntu-2004>...
       Deleting kitchen sandbox at /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Deleting verifier sandbox at /home/uo/.dokken/verifier_sandbox/44d9fc97ad-default-ubuntu-2004
       Finished destroying <default-ubuntu-2004> (0m0.02s).
-----> Testing <default-ubuntu-2004>
-----> Creating <default-ubuntu-2004>...
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
       Creating kitchen sandbox at /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Creating verifier sandbox at /home/uo/.dokken/verifier_sandbox/44d9fc97ad-default-ubuntu-2004
       Building work image..
       Creating container 44d9fc97ad-default-ubuntu-2004
       Finished creating <default-ubuntu-2004> (0m7.36s).
-----> Converging <default-ubuntu-2004>...
       Creating kitchen sandbox in /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Preparing dna.json
       Preparing current project directory as a cookbook
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
       +---------------------------------------------+
       ✔ 2 product licenses accepted.
       +---------------------------------------------+
       Chef Infra Client, version 17.6.18
       Patents: https://www.chef.io/patents
       Infra Phase starting
       [2021-12-01T15:38:41+00:00] ERROR: shard_seed: Failed to get dmi property serial_number: is dmidecode installed?
       Creating a new client identity for default-ubuntu-2004 using the validator key.
       Resolving cookbooks for run list: ["dummy"]
       Synchronizing cookbooks:
         - dummy (0.0.0)
       Installing cookbook gem dependencies:
       Compiling cookbooks...
       Loading Chef InSpec profile files:
       Loading Chef InSpec input files:
       Loading Chef InSpec waiver files:
       Converging 0 resources
       
       Running handlers:
       Running handlers complete
       Infra Phase complete, 0/0 resources updated in 01 seconds
       Finished converging <default-ubuntu-2004> (0m2.94s).
-----> Setting up <default-ubuntu-2004>...
       Finished setting up <default-ubuntu-2004> (0m0.00s).
-----> Verifying <default-ubuntu-2004>...
       Loaded tests from {:path=>".data.code.chef.cookbooks.dummy.test.integration.default"} 

Profile: tests from {:path=>"/data/code/chef/cookbooks/dummy/test/integration/default"} (tests from {:path=>".data.code.chef.cookbooks.dummy.test.integration.default"})
Version: (not specified)
Target:  docker://9f18b81f3f63a1ecfddbffb1bec83a10d872baef2a9eb4b97cc56c028bdcd256

     No tests executed.

Test Summary: 0 successful, 0 failures, 0 skipped
       Finished verifying <default-ubuntu-2004> (0m2.12s).
-----> Destroying <default-ubuntu-2004>...
       Deleting kitchen sandbox at /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Deleting verifier sandbox at /home/uo/.dokken/verifier_sandbox/44d9fc97ad-default-ubuntu-2004
       Finished destroying <default-ubuntu-2004> (0m11.15s).
       Finished testing <default-ubuntu-2004> (0m23.66s).
-----> Test Kitchen is finished. (0m24.62s)

Actual Result

$ DOCKER_HOST=unix:///run/user/1000/podman/podman.sock kitchen test
-----> Starting Test Kitchen (v3.1.1)
-----> Cleaning up any prior instances of <default-ubuntu-2004>
-----> Destroying <default-ubuntu-2004>...
       Deleting kitchen sandbox at /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Deleting verifier sandbox at /home/uo/.dokken/verifier_sandbox/44d9fc97ad-default-ubuntu-2004
       Finished destroying <default-ubuntu-2004> (0m2.55s).
-----> Testing <default-ubuntu-2004>
-----> Creating <default-ubuntu-2004>...
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
       Creating kitchen sandbox at /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Creating verifier sandbox at /home/uo/.dokken/verifier_sandbox/44d9fc97ad-default-ubuntu-2004
       Building work image..
       Creating container 44d9fc97ad-default-ubuntu-2004
       Finished creating <default-ubuntu-2004> (0m11.39s).
-----> Converging <default-ubuntu-2004>...
       Creating kitchen sandbox in /home/uo/.dokken/kitchen_sandbox/44d9fc97ad-default-ubuntu-2004
       Preparing dna.json
       Preparing current project directory as a cookbook
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
       sh: 3: /opt/chef/bin/chef-client: not found
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <default-ubuntu-2004>.  Please see .kitchen/logs/default-ubuntu-2004.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

➕ Additional context

Not sure if this is a bug or I'm doing something wrong...

@UncleOwen
Copy link
Author

Inside the container, /opt/chef, /opt/kitchen and /opt/verifier are completely empty.

@koldrid
Copy link

koldrid commented Dec 15, 2021

This happened to me when the Platform were mixed between the OS container and Chef Container. The version of Chef I was using only had a linux/amd64 version while the OS container was run on linx/arm64

@UncleOwen
Copy link
Author

Interesting. Although in my case, everything is linux/amd64.

@dwmarshall
Copy link

I have been looking into this quite a bit recently! My basic hypothesis has been that this was related to SELinux, but I get the same results when I am running test-kitchen as root.

Looking around in the podman repositories, this issue seems relevant: containers/podman#12638

I have tried fiddling with kitchen-dokken to add a trailing :z to container names, but that hasn't resulted in any success so far.

podman inspect on the container that test-kitchen created seems to have the correct information, but the data isn't there.

@dwmarshall
Copy link

My current belief is that podman's docker API emulation isn't handling VolumesFrom correctly.

@chef-davin
Copy link

So I dug into this a while back but didn't get around to opening up the issue here. There is a workaround for this that you can use. There’s a difference between the ways that docker and podman create containers from images and when a volume gets populated from the image.

In Docker:

  1. docker create --name chef-latest docker.io/chef/chef:latest true will create a container but not start it based on the chef:latest image.
  2. The data for the /opt/chef volume is populated into /var/lib/docker/volumes/<VOLUME_ID>/_data. This is where all chef executable data lives.
  3. kitchen can then use this volume as part of the VolumesFrom field in the docker compose file that gets created for your kitchen images. Since the volume data was populated on container creation, when you kitchen instances start up, they populate /opt/chef with that volume data.

In Podman:

  1. podman create --name chef-latest docker.io/chef/chef:latest true will create a container but not start it, just like with docker.
  2. Unlike with docker. The volume directory is created but NOT populated with data from the image. The volume path is at /var/lib/containers/storage/volumes/<VOLUME_ID>/_data but it is empty.
  3. When kitchen goes to use the VolumesFrom feature in the docker compose file, it does load the volume from the chef-latest container, however, because the container never populated the volume from the image, the volume loaded to /opt/chef on our kitchen instances is empty. Hence you get the error that there is no chef-client executable.

We need the container in Podman to actually run in order to load the data into the volume. It’s important to note that the executable true doesn’t run long enough to fully start the instance and therefore won’t load the data into the volume. You need to use a different executable. I found that sh works.

So, in order to get kitchen-dokken working with Podman, you’ll want to manually create and start the chef-latest or chef-<VERSION> container first before running your kitchen create.

Here’s what I did:

  1. podman create --name chef-latest docker.io/chef/chef:latest sh
  2. podman start chef-latest
  3. verify /opt/chef data populated in /var/lib/containers/storage/volumes/<VOLUME_ID>/_data
  4. run kitchen converge

You CAN do all of that with lifecycle hooks, so I was able to build a kitchen.yml file that accounts for having podman and will ensure that the appropriate chef-latest image was created and run so the volume is populated.

Note, if you’re using a specific version of chef, and not chef-latest, then you need to reference the correct version in your podman create command because this breaks the automatic pulling of the correct version of the chef docker image by kitchen-dokken.

Here’s my kitchen.yml for chef-latest:

---
driver:
  name: dokken
  privileged: true  # allows systemd services to start

provisioner:
  name: dokken

transport:
  name: dokken

verifier:
  name: inspec

platforms:
  # @see https://github.com/chef-cookbooks/testing_examples/blob/main/kitchen.dokken.yml
  # @see https://hub.docker.com/u/dokken
  - name: ubuntu-20.04
    driver:
      image: dokken/ubuntu-20.04
      pid_one_command: /bin/systemd
      intermediate_instructions:
        - RUN /usr/bin/apt-get update

  - name: centos-8
    driver:
      image: dokken/centos-8
      pid_one_command: /usr/lib/systemd/systemd

suites:
  - name: default
    run_list:
      - recipe[test_linux::default]
    verifier:
      inspec_tests:
        - test/integration/default
    lifecycle:
      pre_create:
      - podman create --name chef-latest --replace docker.io/chef/chef:latest sh
      - podman start chef-latest
      post_destroy:
      - podman volume prune -f
    attributes:

As I rethink about this now, you could probably use an environment variable to set the chef-client version you want to use and have that populate things in your kitchen.yml Something like this:

---
driver:
  name: dokken
  privileged: true  # allows systemd services to start
  chef_version: <%= ENV['CHEF_VERSION'] %>

provisioner:
  name: dokken

transport:
  name: dokken

verifier:
  name: inspec

platforms:
  # @see https://github.com/chef-cookbooks/testing_examples/blob/main/kitchen.dokken.yml
  # @see https://hub.docker.com/u/dokken
  - name: ubuntu-20.04
    driver:
      image: dokken/ubuntu-20.04
      pid_one_command: /bin/systemd
      intermediate_instructions:
        - RUN /usr/bin/apt-get update

  - name: centos-8
    driver:
      image: dokken/centos-8
      pid_one_command: /usr/lib/systemd/systemd

suites:
  - name: default
    run_list:
      - recipe[test_linux::default]
    verifier:
      inspec_tests:
        - test/integration/default
    lifecycle:
      pre_create:
      - podman create --name chef-<%= ENV['CHEF_VERSION'] %> --replace docker.io/chef/chef:<%= ENV['CHEF_VERSION'] %> sh
      - podman start chef-<%= ENV['CHEF_VERSION'] %>
      post_destroy:
      - podman volume prune -f
    attributes:

damacus added a commit that referenced this issue Apr 5, 2022
Closes #255

Signed-off-by: Dan Webb <dan.webb@damacus.io>
@damacus damacus mentioned this issue Apr 5, 2022
3 tasks
damacus added a commit that referenced this issue Apr 6, 2022
Closes #255

Signed-off-by: Dan Webb <dan.webb@damacus.io>
@Firefishy
Copy link

Rather than using the lifecycle workaround to fix the mount issue, wouldn't a better solution be to for dokken to change the generated Dockerfile used to create the work image to be a multitage build? Something like this:

FROM ghcr.io/test-kitchen/dokken/debian-12
LABEL X-Built-By=kitchen-dokken X-Built-From=ghcr.io/test-kitchen/dokken/debian-12

COPY --from=docker.io/chef/chef:18 /opt/chef /opt/chef

Notes:

  1. Always use fully registry resolved image names eg: docker.io/chef/chef:18 instead of chef/chef:18 for better podman compatibility. Podman does not default to docker.io registry unlike docker.
  2. Added the COPY with --from= to copy chef into the work image rather than requiring a volume mount.

apenney pushed a commit to apenney/chef that referenced this issue Feb 7, 2025
Due to differences in how podman and docker treat volumes, we have
to hack in a workaround for podman to start the chef container up
before it runs a suite.  Otherwise you get a failure to find the
chef-client binary

test-kitchen/kitchen-dokken#255 has more
information on the cause
apenney pushed a commit to apenney/chef that referenced this issue Feb 7, 2025
Due to differences in how podman and docker treat volumes, we have
to hack in a workaround for podman to start the chef container up
before it runs a suite.  Otherwise you get a failure to find the
chef-client binary

test-kitchen/kitchen-dokken#255 has more
information on the cause
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants