lxd: fails to launch containers #25280

mt-caret · 2017-04-28T07:36:58Z

Issue description

Trying to launch a container produces an error.

$ lxc info --show-log test | grep ERROR

lxc 20170428072645.996 ERROR    lxc_conf - conf.c:instantiate_veth:2669 - failed to attach 'veth3H0S6D' to the bridge 'lxdbr0': Operation not permitted
lxc 20170428072646.944 ERROR    lxc_conf - conf.c:lxc_create_network:2962 - failed to create netdev
lxc 20170428072646.945 ERROR    lxc_start - start.c:lxc_spawn:1088 - Failed to create the network.
lxc 20170428072646.946 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "test"

Steps to reproduce

set virtualisation.lxd.enable = true; and add use to group "lxd", then run $ lxc launch ubuntu:16.04 test

Technical details

System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
17.09pre106117.c90998d5cf (Hummingbird)
Nix version: (run nix-env --version)
nix-env (Nix) 1.11.8
Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
"17.09pre106117.c90998d5cf"
lxd version
2.0.2

The text was updated successfully, but these errors were encountered:

mt-caret · 2017-07-18T19:31:31Z

Solved this a while ago with:

sudo mkdir -p /var/lib/lxc/rootfs
enable apparmor in configuration.nix (Incomplete AppArmor support raphael/linux-samus#103)

I'm not 100% sure if both of these are required or are still relevant, so please keep that in mind.

EDIT: These instructions are misleading/inaccurate. Please refer to #25280 (comment).

fpletz · 2017-07-20T18:19:11Z

We should fix this in the lxd module.

mt-caret · 2017-07-21T10:52:11Z

I would love to, but I don't have much free time right now; salvaging what looks relevant from my configuration.nix gives me

security.apparmor = {
  enable = true;
  profiles = [
    "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start" 
    "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
  ];
  packages = [ pkgs.lxc ];
}

EDIT: fixed typo in code lxc-star -> lxc-start

stefano-m · 2017-12-15T16:21:11Z

Just a note to say that although I have added the configuration suggested in #25280 (comment) I still cannot start containers.

Specifically, lxc tells me that my kernel does not support Apparmor

System information:

nixos-version
17.09.2182.7f6f0c49f0 (Hummingbird)

nix-env --version
nix-env (Nix) 1.11.15

nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
"17.09.2182.7f6f0c49f0"

grep linuxPackages /etc/nixos/hardware-configuration.nix
  boot.kernelPackages = pkgs.linuxPackages_hardened;

uname -r
4.13.16

(but I have also tried unsuccessfully with the stock kernel and the 4.13 one)

# /etc/nixos/lxd.conf
{ config, pkgs, ... }:

{ 
  virtualisation.lxd.enable = true;
  virtualisation.lxc.enable = true;

  # https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-316971140
  security.apparmor = {
    enable = true;
    profiles = [
      "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start"
      "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
    ];
    packages = [ pkgs.lxc ];
  };
}

(I also tried to add virtualisation.lxc.defaultConfig = "lxc.aa_allow_incomplete = 1"; to no avail)

lxc --version
2.16

lxd --version
2.16

Error log

lxc launch images:alpine/3.7/amd64 testcontainer1

lxc info --show-log local:testcontainer1
Name: testcontainer1
Remote: unix://
Architecture: x86_64
Created: 2017/12/15 16:17 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

            lxc 20171215161722.864 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:218 - Incomplete AppArmor support in your kernel
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
            lxc 20171215161722.864 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
            lxc 20171215161722.864 ERROR    lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "testcontainer1".
            lxc 20171215161722.881 ERROR    lxc_conf - conf.c:run_buffer:408 - Script exited with status 1.
            lxc 20171215161722.881 ERROR    lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container "testcontainer1".
            lxc 20171215161722.881 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20171215161722.882 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1253 - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd//lxc/testcontainer1: Device or resource busy
            lxc 20171215161722.882 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/systemd//lxc/testcontainer1
            lxc 20171215161722.882 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1253 - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory//lxc/testcontainer1: Device or resource busy
            lxc 20171215161722.882 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/memory//lxc/testcontainer1

mt-caret · 2018-02-10T04:24:04Z

I reinstalled NixOS on my laptop for an unrelated reason so I was forced to go through setting up lxd again; I'm documenting what works along with links that helped. Hopefully someone with a bit more NixOS expertise can update the module so we don't have to deal with this.

Apologies to @stefano-m; #25280 (comment) was kind of misleading.

create missing directory: sudo mkdir -p /var/lib/lxc/rootfs

This will give you errors as shown here: #25280 (comment) and raphael/linux-samus#103

allow incomplete apparmor lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 as shown here: Incomplete AppArmor support raphael/linux-samus#103 (comment)

This will now give you these messages as shown here: lxc/lxc#1895

lxc 20180210034825.988 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:218 - Incomplete AppArmor support in your kernel
lxc 20180210034825.988 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-container-default-cgns

The lxc issue in question has a comment with a workaround that seems to involve disabling apparmor altogether, but this can also be fixed by adding the relevant apparmor profiles (I can't remember where or how I found this information).

add profiles for lxc in configuration.nix

security.apparmor = {
  enable = true;
  profiles = [
    "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start"
    "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
  ];
  packages = [ pkgs.lxc ];
};

Now lxc launch ubuntu:16.04 should work without any problems.

mt-caret · 2018-02-19T13:49:01Z

Now #34803 has been merged, I believe all you need to do to get lxd up and running is to enable lxd in configuration.nix, lxd init, and lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 AFAIK (would love if someone can confirm this).

mt-caret · 2018-05-15T01:21:56Z

This issue seems relevant: https://github.com/lxc/lxd/issues/2834

mt-caret · 2018-11-10T12:23:58Z

As of 18.09, lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 no longer seems to be necessary.

mt-caret · 2018-11-10T12:25:23Z

AFAIK this is now solved (virtualisation.lxd.enable = true will now work without additional effort) so closing.

mt-caret closed this as completed Jul 18, 2017

fpletz reopened this Jul 20, 2017

fpletz added this to the 17.09 milestone Jul 20, 2017

fpletz added 0.kind: bug Something is broken 3.skill: sprintable A larger issue which is split into distinct actionable tasks labels Jul 20, 2017

tad-lispy mentioned this issue Aug 5, 2017

Running LXD > 2.3 on NixOS #27967

Closed

CosmicToast mentioned this issue Nov 2, 2017

LXD should have more tools in its path #31117

Closed

6 tasks

mt-caret mentioned this issue Feb 10, 2018

nixos/lxd: partial fix for lxd breakage #34803

Merged

8 tasks

matthewbauer modified the milestones: 17.09, 18.09 Apr 17, 2018

matthewbauer modified the milestones: 18.09, 19.03 Nov 5, 2018

mt-caret closed this as completed Nov 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lxd: fails to launch containers #25280

lxd: fails to launch containers #25280

mt-caret commented Apr 28, 2017

mt-caret commented Jul 18, 2017 •

edited

Loading

fpletz commented Jul 20, 2017

mt-caret commented Jul 21, 2017 •

edited

Loading

stefano-m commented Dec 15, 2017

mt-caret commented Feb 10, 2018 •

edited

Loading

mt-caret commented Feb 19, 2018 •

edited

Loading

mt-caret commented May 15, 2018 •

edited

Loading

mt-caret commented Nov 10, 2018

mt-caret commented Nov 10, 2018

lxd: fails to launch containers #25280

lxd: fails to launch containers #25280

Comments

mt-caret commented Apr 28, 2017

Issue description

Steps to reproduce

Technical details

mt-caret commented Jul 18, 2017 • edited Loading

fpletz commented Jul 20, 2017

mt-caret commented Jul 21, 2017 • edited Loading

stefano-m commented Dec 15, 2017

System information:

Error log

mt-caret commented Feb 10, 2018 • edited Loading

mt-caret commented Feb 19, 2018 • edited Loading

mt-caret commented May 15, 2018 • edited Loading

mt-caret commented Nov 10, 2018

mt-caret commented Nov 10, 2018

mt-caret commented Jul 18, 2017 •

edited

Loading

mt-caret commented Jul 21, 2017 •

edited

Loading

mt-caret commented Feb 10, 2018 •

edited

Loading

mt-caret commented Feb 19, 2018 •

edited

Loading

mt-caret commented May 15, 2018 •

edited

Loading