Buildkit does not configure DNS correctly inside container #1004

abergmeier · 2019-05-13T15:14:35Z

For a project, I enabled DOCKER_BUILDKIT="1".
When building, I get wget: bad address 'my.company.com'.

The offending line is:

RUN wget http://my.company.com/foobar.txt

I am not really sure, why this does not work. Other lines like apk add --no-cache git work fine in contrast.

When experimenting, I get a similar error with:

RUN go mod download

And this works fine both on the host and when not using Buildkit.

The text was updated successfully, but these errors were encountered:

AkihiroSuda · 2019-05-13T17:03:22Z

Docker version?

tonistiigi · 2019-05-13T17:27:45Z

Any custom dns config?

abergmeier · 2019-05-13T18:35:52Z

Any custom dns config?

Not that I know of.

abergmeier · 2019-05-14T07:02:48Z

Docker version?

docker --version
Docker version 18.09.5, build e8ff056

On Ubuntu Bionic

abergmeier · 2019-05-14T14:31:54Z

Looking at journalctl I see the following entries:

systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
systemd-resolved[1024]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
dockerd[2070]: time="2019-05-14T14:24:12Z" level=error msg="fatal error: rpc error: code = Unknown desc = executor failed running [/bin/sh -c wget http://my.company.com/foobar.txt]: exit code: 1"
dockerd[2070]: panic: rpc error: code = Unknown desc = executor failed running [/bin/sh -c wget http://my.company.com/foobar.txt]: exit code: 1
dockerd[2070]: goroutine 1 [running]:
dockerd[2070]: main.main()
dockerd[2070]:         /src/frontend/dockerfile/cmd/dockerfile-frontend/main.go:26 +0x34f

abergmeier · 2019-05-14T14:57:23Z

cat /etc/resolv.conf on host prints:

nameserver 127.0.0.53
options edns0

so this might be related to moby/libnetwork#1654!?

In the container it prints:

options edns0
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

which seems to miss the local and company DNS Servers.

tonistiigi · 2019-05-14T17:59:28Z

@abergmeier Just so that everything is clear, can you confirm that you do not have connectivity to 8.8.8.8 nameserver from this machine. In the output you posted about "In the container it prints:", is this from docker run or from RUN cat /etc/resolv.conf in Dockerfile. If not latter can you post that as well and can you confirm that you do have network connectivity from regular containers run with docker run.

abergmeier · 2019-05-14T18:29:26Z

from RUN cat /etc/resolv.conf in Dockerfile

That.

tiborvass · 2019-05-15T00:09:09Z

Apologies @abergmeier but we're trying to zero in this issue and I'm a bit confused right now.
For sake of completeness and being extra sure we're not missing anything, do you mind providing inline outputs for the following:

ls -l /etc/resolv.conf
systemctl status systemd-resolved.service
cat /etc/docker/daemon.json | grep -i dns
docker run -it --rm alpine cat /etc/resolv.conf
echo -e "from alpine\n run cat /etc/resolv.conf" | DOCKER_BUILDKIT=0 docker build --no-cache --progress plain
echo -e "from alpine\n run cat /etc/resolv.conf" | DOCKER_BUILDKIT=1 docker build --no-cache --progress plain
ps aux | grep dockerd

Thanks!

abergmeier · 2019-05-15T07:18:10Z

There you go:

Output: lrwxrwxrwx 1 root root /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
Output:

● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-05-15 09:02:10 CEST; 5min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 1000 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/systemd-resolved.service
           └─1000 /lib/systemd/systemd-resolved

Mai 15 09:02:46 s0121552 systemd-resolved[1000]: Using degraded feature set (UDP) for DNS server 8.8.4.4.
Mai 15 09:02:47 s0121552 systemd-resolved[1000]: Using degraded feature set (UDP) for DNS server 8.8.8.8.
Mai 15 09:02:52 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:02:56 s0121552 systemd-resolved[1000]: Using degraded feature set (TCP) for DNS server 8.8.8.8.
Mai 15 09:02:56 s0121552 systemd-resolved[1000]: Using degraded feature set (TCP) for DNS server 8.8.4.4.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:03 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:19 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mai 15 09:03:19 s0121552 systemd-resolved[1000]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

Output: cat: /etc/docker/daemon.json: No such file or directory
Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

Output:

options edns0

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

which seems pretty wrong, especially since our network does not route ipv6!

Output: root 2084 1.2 0.3 5676068 123720 ? Ssl 09:02 0:11 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

abergmeier · 2019-05-15T07:26:29Z

Sadly I cannot seem to override /etc/resolv.conf inside the container, so I cannot work around this 😢

thaJeztah · 2019-05-15T07:48:58Z

searching for that error, I came to this page https://www.linode.com/community/questions/17384/error-server-returned-error-nxdomain-mitigating-potential-dns-violation-dve-2018

which indicates this may be a bug in systemd resolvd (just from a quick read)

abergmeier · 2019-05-15T09:39:42Z

which indicates this may be a bug in systemd resolvd (just from a quick read)

This is probably one factor. On the other hand, Buildkit IMO definitely has a bug where it handles resolv.conf different than legacy Docker (with the latter working fine).

tiborvass · 2019-05-15T18:13:40Z

@abergmeier Thanks, can you also provide the output for cat /run/systemd/resolve/resolv.conf please ?

abergmeier · 2019-05-16T13:53:05Z

cat /run/systemd/resolve/resolv.conf outputs:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

tiborvass · 2019-06-04T05:10:43Z

@abergmeier would you mind trying a dockerd binary I built from moby/moby#39295 ?

Here's a link: https://drive.google.com/file/d/15hJWKtlEUpoqYi7uKsaO20qqtEREZ768/view?usp=sharing
The sha256 is c7ed7bb6011ce6fd3270d69d528a01a18060a30074028ca1dca334299d3b2c6f.

abergmeier · 2019-06-04T09:27:35Z

Seems to fix the problem 🎉
6. Output:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.79.255.100
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.79.255.200

varlucian · 2019-06-28T10:55:39Z

Will the issue be backported to 18.06?
If not, what is the workaround?

thaJeztah · 2019-06-28T10:57:25Z

18.06 is EOL, so it won't be backported to that version

varlucian · 2019-06-28T12:50:43Z

Sorry, I meant 18.09 not 18.06

abdennour · 2023-11-27T18:26:07Z

same issue with rootless dind

abergmeier changed the title ~~BUILDKIT~~ Buildkit cannot access network May 13, 2019

abergmeier changed the title ~~Buildkit cannot access network~~ Buildkit does not configure DNS correctly inside container May 15, 2019

This was referenced May 31, 2019

executor: use systemd resolvconf if enabled #1033

Merged

build: buildkit now also uses systemd's resolv.conf moby/moby#39295

Merged

tiborvass closed this as completed in #1033 Jun 4, 2019

thaJeztah mentioned this issue Jun 4, 2019

[19.03 backport] build: buildkit now also uses systemd's resolv.conf docker-archive/engine#260

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildkit does not configure DNS correctly inside container #1004

Buildkit does not configure DNS correctly inside container #1004

abergmeier commented May 13, 2019

AkihiroSuda commented May 13, 2019

tonistiigi commented May 13, 2019

abergmeier commented May 13, 2019

abergmeier commented May 14, 2019 •

edited

Loading

abergmeier commented May 14, 2019

abergmeier commented May 14, 2019 •

edited

Loading

tonistiigi commented May 14, 2019

abergmeier commented May 14, 2019

tiborvass commented May 15, 2019 •

edited

Loading

abergmeier commented May 15, 2019 •

edited

Loading

abergmeier commented May 15, 2019

thaJeztah commented May 15, 2019

abergmeier commented May 15, 2019

tiborvass commented May 15, 2019

abergmeier commented May 16, 2019

tiborvass commented Jun 4, 2019

abergmeier commented Jun 4, 2019

varlucian commented Jun 28, 2019

thaJeztah commented Jun 28, 2019

varlucian commented Jun 28, 2019

abdennour commented Nov 27, 2023

Buildkit does not configure DNS correctly inside container #1004

Buildkit does not configure DNS correctly inside container #1004

Comments

abergmeier commented May 13, 2019

AkihiroSuda commented May 13, 2019

tonistiigi commented May 13, 2019

abergmeier commented May 13, 2019

abergmeier commented May 14, 2019 • edited Loading

abergmeier commented May 14, 2019

abergmeier commented May 14, 2019 • edited Loading

tonistiigi commented May 14, 2019

abergmeier commented May 14, 2019

tiborvass commented May 15, 2019 • edited Loading

abergmeier commented May 15, 2019 • edited Loading

abergmeier commented May 15, 2019

thaJeztah commented May 15, 2019

abergmeier commented May 15, 2019

tiborvass commented May 15, 2019

abergmeier commented May 16, 2019

tiborvass commented Jun 4, 2019

abergmeier commented Jun 4, 2019

varlucian commented Jun 28, 2019

thaJeztah commented Jun 28, 2019

varlucian commented Jun 28, 2019

abdennour commented Nov 27, 2023

abergmeier commented May 14, 2019 •

edited

Loading

abergmeier commented May 14, 2019 •

edited

Loading

tiborvass commented May 15, 2019 •

edited

Loading

abergmeier commented May 15, 2019 •

edited

Loading