[WIP] Support advanced mounts syntax

[shin-]

Fixes #5181
Fixes #5244

[StefanScherer]

Tried commit a594bef with

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - C:\\Users\\stefan\\.ssh:C:\\ssh

networks:
  default:
    external:
      name: nat

but still see this error, both on a Windows host as well as remote from a Mac against a Windows Docker host.

Creating sshkey_test_1 ... error

ERROR: for sshkey_test_1  Cannot create container for service test: invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\\ssh:rw'

ERROR: for test  Cannot create container for service test: invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\\ssh:rw'

[shin-]

Can you try with version 3.4 instead?

[StefanScherer]

With version 3.4 it shows

Creating sshkey_test_1 ... error

ERROR: for sshkey_test_1  Cannot create container for service test: invalid mount config for type "bind": invalid mount path: 'C:\\ssh'

ERROR: for test  Cannot create container for service test: invalid mount config for type "bind": invalid mount path: 'C:\\ssh'
ERROR: Encountered errors while bringing up the project.

Both with a Windows docker-compose.exe and a Mac docker-compose binary.

[shin-]

Okay, thanks for taking the time to check this, really appreciate it =)
I'll keep working on it to make sure those cases are supported.

[shin-]

Closing in favor of #5436

[StefanScherer]

Should this work with 1.18.0-rc1? It doesn't work for me for windows paths and the npipe.

[shin-]

Are you writing it like this?

volumes:
  - type: npipe
    source: \\.\pipe\pipename
    target: ...

It won't work with the short syntax.

For the Windows paths from a UNIX client, unfortunately I had to postpone it. It's more intricate than I originally thought. I'm hoping to have a reasonable solution for it in 1.19.

[StefanScherer]

I have tried it with the Windows docker-compose.exe. I‘ll try your sugested way.

[StefanScherer]

Thanks. This works:

version: "3.5"

services:
  test:
    image: stefanscherer/docker-cli-windows
    volumes:
      - type: npipe
        source: \\.\pipe\docker_engine
        target: \\.\pipe\docker_engine
      - type: bind
        source: C:\Users\stefan\.ssh
        target: C:\ssh
    command: docker version

networks:
  default:
    external:
      name: nat

[StefanScherer]

It also works from a Mac running a Windows Server 1709 VM with same VM mount points ( /Users/stefan = C:\Users\stefan in the Windows VM ). 💯

version: "3.5"

services:

  dockercli:
    image: stefanscherer/docker-cli-windows
    volumes:
      - type: npipe
        source: \\.\pipe\docker_engine
        target: \\.\pipe\docker_engine
    command: docker version

  sshcli:
    image: microsoft/nanoserver:1709
    volumes:
      - type: bind
        source: C:\Users\stefan\.ssh
        target: C:\ssh
    command: cmd /c dir c:\ssh

networks:
  default:
    external:
      name: nat

$ env | grep DOCK
DOCKER_HOST=tcp://192.168.65.218:2376
DOCKER_MACHINE_NAME=1709
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/stefan/.docker/machine/machines/1709

$ docker-compose up
Creating npipe_sshcli_1    ... done
Creating npipe_dockercli_1 ... done
Attaching to npipe_dockercli_1, npipe_sshcli_1
sshcli_1     |  Volume in drive C has no label.
sshcli_1     |  Volume Serial Number is 586A-90FC
sshcli_1     | 
sshcli_1     |  Directory of C:\ssh
sshcli_1     | 
sshcli_1     | 11/28/2017  11:16 PM    <DIR>          .
sshcli_1     | 12/07/2017  12:14 AM    <DIR>          ..
sshcli_1     | 10/22/2015  11:48 AM               483 config
sshcli_1     | 06/22/2015  05:17 AM             1,766 id_rsa
sshcli_1     | 06/22/2015  05:17 AM               408 id_rsa.pub
sshcli_1     | 12/06/2017  03:17 PM            43,746 known_hosts
dockercli_1  | Client:
dockercli_1  |  Version:      17.10.0-ce
dockercli_1  |  API version:  1.33
dockercli_1  |  Go version:   go1.8.3
dockercli_1  |  Git commit:   f4ffd25
dockercli_1  |  Built:        Tue Oct 17 19:00:02 2017
dockercli_1  |  OS/Arch:      windows/amd64
dockercli_1  | 
dockercli_1  | Server:
dockercli_1  |  Version:      17.10.0-ee-preview-3
dockercli_1  |  API version:  1.34 (minimum version 1.24)
dockercli_1  |  Go version:   go1.8.4
dockercli_1  |  Git commit:   b8571fd
dockercli_1  |  Built:        Fri Oct  6 18:01:48 2017
dockercli_1  |  OS/Arch:      windows/amd64
dockercli_1  |  Experimental: false
npipe_dockercli_1 exited with code 0
sshcli_1     |                2 Dir(s)  92,883,705,856 bytes free
npipe_sshcli_1 exited with code 0

[Getitdan]

@StefanScherer I get this error:

ERROR: for traefik Cannot create container for service traefik: b'Mount denied:\nThe source path "\\\\.\\pipe\\docker_engine:\\\\.\\pipe\\docker_engine"\nis not a valid Windows path'
ERROR: Encountered errors while bringing up the project.

It seems it's not possible to use this \\.\pipe\docker_engine source path on a Windows host. How did you succeeded in doing so?

(By the way, I'm running Traefik:)

version: '3.2'
services:
  traefik:
    image: traefik # The official Traefik docker image
    command: --api --docker
    ports:
      - "80:80"     # The HTTP port
      - "8080:8080" # The Web UI (enabled by --api)
    volumes:
      - type: npipe
        source: \\.\pipe\docker_engine
        target: \\.\pipe\docker_engine

[StefanScherer]

@Getitdan it depends on the windows host, the Docker engine version and also the Docker images used.
It starts working with Windows Server 1709, you can use 1803 or the 2019 Insider builds as host or a current Windows 10 with eg. the April update.
Docker engine 18.03.1-ee should work fine with it. I uswd a 17.10 preview version at that time.
TL/DR in the near future it will just work 😀

[Getitdan]

@StefanScherer Thanks a lot for your response. I'm using Windows 10 pro 64 bit, fully updated (no updated are pending, so I guess I have the "April update", no?). My Docker for Windows is also the latest: 18.06.0-ce-win70 (19075). So the only option is to use Insider builds?

My server OS is ubuntu, but my development computer is Windows and I wanted to run docker locally, before deploying it to the Ubuntu server. What do other developers do? No one runs docker locally on Windows 10? I guess not everyone has a Mac OS / Linux.

My temporary workaround is to use a shared folder with a virtualbox running Ubuntu, and there to run docker properly for development. But it is not preferable. I simply can't understand what do other developers with Windows 10 do.

[StefanScherer]

@Getitdan Now I understand you want to bind the named pipe from the Windows host into a Linux container unix socket.
I think this is not supported by the engine yet.

[Getitdan]

@StefanScherer Thanks again! I'll keep using my workaround using a staging VirtualBox.

But just out of curiosity, what are other uses for using named pipe in docker? Isn't it always for binding from a host into a container?

And second thing: didn't you mean connecting named pipe with a named pipe on Linux? Because I guess it should be equivalent (don't we need named pipes on both host & guest, and eliminate unix socket on the latter?). Thanks a lot!

[shin-]

@Getitdan if you're just running Linux containers on Windows and need to access the Docker daemon for some reason, you want to mount /var/run/docker.sock. which will let you connect through the UNIX socket inside the hypervisor (thin virtual machine) that is actually running Docker on your machine.

You can't use named pipes inside Linux containers because named pipes as a concept are Windows-specific.

HTH

[Getitdan]

Thanks @shin, you are right. I'm just running Linux containers on Windows, but cannot access the Docker daemon. I cannot mount /var/run/docker.sock at all. Using a mounting configuration like this:

volumes:
      - ./docker.sock:/var/run/docker.sock

traefik_1    | time="2018-07-27T14:14:07Z" level=error msg="Failed to retrieve information of the docker client and server host: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
traefik_1    | time="2018-07-27T14:14:07Z" level=error msg="Provider connection error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?, retrying in 3.787817738s"

I thought it happens because my Windows host cannot handle the unix socket, so I tried to use a named pipe instead. Doesn't Linux support named pipes? I think it uses the mkfifo command to apply exactly that.

How would you suggest me to "wire" my volume configuration?

Update:
Solved by a workaround stated in: docker/for-win#1829
This issue caused me lots of troubles.