Docker on Windows is not usable with volumes mapped to local directories #394

denisvmedia · 2017-01-17T15:46:45Z

Expected behavior

Remote operations should work as if the directories are local to the container

Actual behavior

There are multiple problems:

It is EXTREMELY slow (for example I can't work normally with Symfony through Docker - cache/vendors folders mapped to the local host are created/updated too slow).
Apart it is too slow, in most of the cases I even can't delete a cache folder from the container (even using rm -rf folder_name). It usually stops at some point saying that the folder is not empty. Rerunning the command may help or may stuck again on another folder. Needless to say that from the php script the things are getting even worse.
Sometimes the files are just getting stuck in the "pending delete state" so that you can't delete it at all (even on the host Windows machine). So, I have to restart docker (or even Windows) to release the lock.

Information

Windows 10 Pro Insider Preview (Build 14986)
Docker Version: 1.12.6 (9655)
Diagnostic ID: D77FC819-C847-424F-8163-5274ED2DE0CD/2017-01-17_16-45-04

Steps to reproduce the behavior

Install a Symfony application to a host folder.
Inside a docker container (can be a php one) try to initialize the app, fill the cache, then try to clean in by different ways.
"Enjoy" the result.

Misc

Until it's solved, Docker is completely unusable for me on Windows (and I don't get how people use it - probably it's ok for really small microservice applications, but for something bigger it's just failing to work properly).

simonferquel · 2017-01-18T17:00:16Z

Hi @denisvmedia, could you try with latest beta ?

reypm · 2017-01-20T03:59:13Z

@denisvmedia I am not experimenting such behavior and I am working with Symfony, maybe I misunderstood the issue. Anyway this is my repository give it a try and let me know if you still experimenting the same issue. I am using the latest beta - a RC I think?

denisvmedia · 2017-01-21T19:00:11Z

@reypm @simonferquel

My research shows that I "reynierpm/docker-centos7-supervisord:latest" image (used in Dockerfile by @reypm) has no problems with deleting the files and folders but "alpine:edge" (which is used by my container) has this problem.

I went further. I took my huge cache folder and put it to d:\data\dev. I ran two commands over it (both times I restored the folder of course).

First test using alpine:latest image:

D:\data>docker run -v d:/data:/data alpine rm -rf /data/dev
rm: can't remove '/data/dev/annotations': Directory not empty
rm: can't remove '/data/dev/assetic/config/0': Directory not empty
rm: can't remove '/data/dev/assetic/config/1': Directory not empty
rm: can't remove '/data/dev/assetic/config/2': Directory not empty
rm: can't remove '/data/dev/assetic/config/3': Directory not empty
rm: can't remove '/data/dev/assetic/config/4': Directory not empty
rm: can't remove '/data/dev/assetic/config/5': Directory not empty
rm: can't remove '/data/dev/assetic/config/6': Directory not empty
rm: can't remove '/data/dev/assetic/config/8': Directory not empty
rm: can't remove '/data/dev/assetic/config/9': Directory not empty
rm: can't remove '/data/dev/assetic/config/a': Directory not empty
rm: can't remove '/data/dev/assetic/config/b': Directory not empty
rm: can't remove '/data/dev/assetic/config/d': Directory not empty
rm: can't remove '/data/dev/assetic/config/f': Directory not empty
rm: can't remove '/data/dev/assetic/config': Directory not empty
rm: can't remove '/data/dev/assetic': Directory not empty
rm: can't remove '/data/dev/translations': Directory not empty
rm: can't remove '/data/dev/twig': Directory not empty
rm: can't remove '/data/dev': Directory not empty

I can run this command multiple times and finally all the files and folders will be deleted.

Second test using reynierpm/docker-centos7:latest image:

D:\data>docker run -v d:/data:/data reynierpm/docker-centos7-supervisord rm -rf /data/dev
2017-01-21 18:58:57,053 CRIT Supervisor running as root (no user in config file)
2017-01-21 18:58:57,053 WARN No file matches via include "/etc/supervisor.d/*.conf"
2017-01-21 18:58:57,065 INFO RPC interface 'supervisor' initialized
2017-01-21 18:58:57,066 INFO supervisord started with pid 9

D:\data>docker run -v d:/data:/data reynierpm/docker-centos7-supervisord ls -lia /data/dev
ls: cannot access /data/dev: No such file or directory

How come it works with one image and doesn't work the the other?

denisvmedia · 2017-01-21T19:06:13Z

Here is another example that works. So, seems there is an issue only with alpine...

D:\data>docker run -v d:/data:/data centos ls -lia /data
total 4
7881299348246725 drwxr-xr-x  2 root root    0 Jan 21 19:04 .
               2 drwxr-xr-x 23 root root 4096 Jan 21 19:04 ..
6473924465068401 -rwxr-xr-x  1 root root    0 Jan 21 18:49 1.txt
 562949954166349 drwxr-xr-x  2 root root    0 Jan 21 18:54 conf
4503599628108512 drwxr-xr-x  2 root root    0 Jan 21 19:04 dev
 562949954166351 drwxr-xr-x  2 root root    0 Jan 21 18:54 logs
 562949954166350 drwxr-xr-x  2 root root    0 Jan 21 18:54 run

D:\data>docker run -v d:/data:/data centos rm -rf /data/dev

D:\data>docker run -v d:/data:/data centos ls -lia /data
total 4
7881299348246725 drwxr-xr-x  2 root root    0 Jan 21 19:05 .
               2 drwxr-xr-x 23 root root 4096 Jan 21 19:05 ..
6473924465068401 -rwxr-xr-x  1 root root    0 Jan 21 18:49 1.txt
 562949954166349 drwxr-xr-x  2 root root    0 Jan 21 18:54 conf
 562949954166351 drwxr-xr-x  2 root root    0 Jan 21 18:54 logs
 562949954166350 drwxr-xr-x  2 root root    0 Jan 21 18:54 run

reypm · 2017-01-21T22:41:01Z

@denisvmedia can you share your Dockerfile?

denisvmedia · 2017-01-21T22:45:22Z

@reypm it happens even without Dockerfile... This command docker run -v d:/data:/data alpine rm -rf /data/dev runs with errors if /data/dev is fairly big (e.g. cache dir from Symfony). And it happens only with alpine image. I switched to centos:latest and the problem is gone.

denisvmedia · 2017-01-22T10:18:26Z

However, the problem with the files is gone, still the productivity is bad... The app loading too slow. And I don't know how to prove that (as I'm not allowed to share the sources). It takes 13 seconds to load the app. And assets in dev-mode are being loaded forever :( On the same PC without Docker it works fast.

If anyone knows an open source Symfony app with reach functionality that I could test in Docker, I could prepare a proof of the problem.

cfstras · 2017-03-03T12:15:12Z

This issue also occurs on my environment, mainly when compiling programs on a shared volume.
Docker seems to create files with stale handles, which cannot be removed until the Docker VM is stopped.
If you try to delete the folder from windows, you get "You need permission to perform this action". Deleting the file seems to work, however it will reappear when you reload.

After clicking "Continue":

Here is an example to recreate.
Basically, I start a centos:7 container, install bzip2 and download alsa-lib sources.
Then, I rapidly extract them and delete them again, interrupting the process. This leads to the corruption: as soon as I try it again, I hit that invalid file.

[claus@windows]$ mkdir test2
[claus@windows]$ docker run -it --rm -v $(pwd)/test2:/test2 centos:7 bash
[root@7c4e4f8ab238 /]# yum -y install bzip2
... yada yada...
[root@7c4e4f8ab238 /]# cd test2
[root@7c4e4f8ab238 /]# curl -O "ftp://ftp.alsa-project.org/pub/lib/alsa-lib-1.1.3.tar.bz2"
[root@7c4e4f8ab238 test2]# while true; do tar xf alsa-lib-1.1.3.tar.bz2  && rm -rf alsa-lib-1.1.3 && echo -n "."; done
................^C
[root@7c4e4f8ab238 test2]# while true; do tar xf alsa-lib-1.1.3.tar.bz2  && rm -rf alsa-lib-1.1.3 && echo -n "."; done
rm: cannot remove 'alsa-lib-1.1.3': Directory not empty
tar: alsa-lib-1.1.3/alsalisp: Cannot mkdir: No such file or directory
tar: alsa-lib-1.1.3/alsalisp: Cannot mkdir: No such file or directory
tar: alsa-lib-1.1.3/alsalisp/alsalisp.c: Cannot open: No such file or directory
tar: alsa-lib-1.1.3/alsalisp: Cannot mkdir: No such file or directory
tar: alsa-lib-1.1.3/alsalisp/Makefile.in: Cannot open: No such file or directory
tar: alsa-lib-1.1.3/alsalisp: Cannot mkdir: No such file or directory
tar: alsa-lib-1.1.3/alsalisp/Makefile.am: Cannot open: No such file or directory
^C
[root@7c4e4f8ab238 test2]#

The workaround so far has been for me to always use unshared volumes, the container's filesystem, or --tmpfs instead of shared volumes. However, it can be pretty annoying to have "rsync -a /volume/ /tmp/volume" in a build script.

I'm using version 17.03.0-ce-win1 (10296) Channel: stable e5a07a1, however this occurred on all past versions I have tried so far.

colinfindlay-nz · 2017-03-23T07:11:11Z

This has been driving me insane all day. I'm seeing it on the node:7 image, which is based off jessie - so not strictly an Apline issue I think.

ezynda3 · 2017-03-23T16:58:23Z

Same issue. I can't use webpack to recompile and version assets because it always complains about deleting a file that doesn't exist. The file shows up inside the image and on the windows host but won't actually complete deleting until I stop the container.

ovecka · 2017-04-11T15:43:14Z

Same issue here with windows + php-fpm + nette framework which is doing caching - fairly similar to symfony caching.
But in our work only I have frequent issues, one has it sometimes after sleep, and one never ... :( Not able to do steps to reproduce but tell me what I should run and I'll try :)

cfstras · 2017-04-13T16:30:26Z

Still happening on version 17.03.1-ce-win5 b18e2a5.

Also, I don't know if this is related: Sometimes, single files fail to be removed. Sometimes, I/O stops working completely after a few minutes of being idle:

[root@56b1c0633d67]# make
make: stat: GNUmakefile: Input/output error
make: stat: makefile: Input/output error
make: stat: Makefile: Input/output error
make: *** No targets specified and no makefile found.  Stop.

vanko-dev · 2017-06-08T05:41:33Z

It really works bad. I was trying to install Jenkins with mapping and was observing random Jenkins's plugins errors and either "yarn install" or "npm install" worked incorrectly. They cannot install all packages inside a volume

Docker version 17.03.1-ce, build c6d412e
image: Linux4.9.27-moby
win 10.0.14393

YaronBasson · 2017-07-02T13:17:04Z

Are there any plans to fix this ? (happens also with 17.06.0-ce-win18 (12627) )

kumarharsh · 2017-11-10T08:19:22Z

+1. It's so slow. Running yarn install takes ages - it copies about 100 files per second from it's cache into the project's node_modules. In Windows proper, it's about 1500 per second.

philippslang · 2017-12-04T11:46:11Z

Similar issue (stale file handles) trying to build with make from within Docker container on a mapped directory. Only when using parallel builds (-j), serial fine.

andreikojusko · 2018-01-11T07:59:37Z

I have nginx/php-fpm in a container, and 2 copies of the same Symfony project:

on shared volume
copied to a local path in the contaier
both are configured in nginx with different URLs.
2nd (local copy) works at least 100 times faster.

Celoain · 2018-01-29T16:26:11Z

Setup a docker with shared volume. php7.0-fpm with nginx, having magento. Loads way too slow.

dashesy · 2018-05-08T22:09:11Z

make -j is slow to the point of being useless inside docker-for-windows. It was fine in the previous version. When I look at the Task Manager, the shared drive utilization is at 100% !! Only after I exit the docker, it goes down.

denisvmedia · 2018-05-20T16:29:48Z

Are the any plans to fix the problem?

dashesy · 2018-05-21T16:42:35Z

@denisvmedia Try removing the Docker-for-Windows, and all the containers, then reinstalling it. It made it a little better for me. Do not re-use the previous images.

docker-robott · 2018-08-19T04:01:11Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

jpierront · 2018-08-19T09:50:21Z

/remove-lifecycle stale

jonasschumacher · 2018-12-06T16:45:09Z

A workaround instead rebooting Windows/Docker would be:

Open Computer Management;
Navigate to Shared Folders > Open Files;
Right click anywhere in the open files area and choose "Disconnect All Open Files".

kumarharsh · 2018-12-10T09:39:00Z

Please look into this issue. This has become a near daily occurrence while using Docker for Windows.

technatelogy · 2018-12-10T18:37:36Z

The workaround only deals with the stale file handle issue. I rarely get the stale file handle issue, the failure to delete issue happens all the time when running anything that quickly deletes and adds new files (i.e. composer, npm, etc.) and requires that you either have to clean the folder yourself manually in windows or run the program repeatedly until it succeeds (neither are great solutions). The main issue appears to just be file system performance. As mentioned before performance is at least 15x slower than native windows.

SC7639 · 2019-01-16T07:47:47Z

I am having similar issues when using npm install inside a container with a mounted volume. If always fails at creating files or removing them. It would be very handy for to mount a folder inside WSL as it would be a Linux filesystem that is operating on.

robertdumitrescu · 2019-02-25T11:15:04Z

Such a shame that no one adresses this. We have an enterprise application which is consisting of 12 mini apps. Every time we run another one in parallel, seems to become exponentially slower. For the 5th app, it took 1 min 20 seconds to sync everything inside node modules. Thats why there are so few devs on Win.

grexlort · 2019-03-29T14:36:25Z

This annoying shit should be fixed ASAP, Symfony dev here - working on windows docker even with hyper-v is nightmare, but i'm to lazy to migrate to unix 🐧

kumarharsh · 2019-03-30T06:18:44Z

See if you can use WSL - it's proving to be much better for me than Docker for Windows.

chrisdfrey · 2019-04-30T14:39:49Z

I have a script that tries to rm -rf a git repo inside a container; it always fails on Docker for Windows.

zjcnew · 2019-09-11T07:42:05Z

A workaround instead rebooting Windows/Docker would be:

Open Computer Management;

Navigate to Shared Folders > Open Files;

Right click anywhere in the open files area and choose "Disconnect All Open Files".

Is it so cumbersome to modify every file?

docker-robott · 2019-12-10T01:00:22Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

sylvainar · 2019-12-10T19:04:21Z

/remove-lifecycle stale

Still struggling with this on an everyday basis

robertdumitrescu · 2020-01-03T23:26:32Z

/remove-lifecycle stale
Still an issue. Definitely not stale

docker-robott · 2020-04-02T01:00:22Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

nverwer · 2020-04-02T07:13:50Z

/remove-lifecycle stale

Burgito · 2020-06-11T13:25:50Z

I have been experimenting this issue a lot with the latest stable builds (2.3.0.2, 2.3.0.3)
I've downgraded my docker desktop for windows installation to 2.1.0.5 (last version using samba) and it doesn't seem to happen anymore.

I don't really understand what is happening but this is a very-hard-to-live-with issue :(

technatelogy · 2020-06-11T21:09:59Z

FYI: This is a non-issue with WSL2. I highly recommend upgrading to Windows 10 2004 release, switching to WSL2 and using Docker's WSL2 integration. Anything with file system intensive operations, composer, npm, etc. are all much faster and actually finish without error.

slominskir · 2020-07-20T17:44:11Z

This is a non-issue with WSL2

WSL2 is not a perfect fix. As described in the Best Practices section of the docs you must put your code inside the WSL VM if you want changes to the files to be detected. This makes working with your favorite IDE or other Windows tools more complicated. More info:

Updates of files outside the container are not visible inside of it #7276
[WSL2] Mounting linux path as volume not working #4812
[WSL2] File changes made by Windows apps on Windows filesystem don't trigger notifications for Linux apps microsoft/WSL#4739
File contents differs on mounted Windows drive between Windows and WSL microsoft/WSL#4064

technatelogy · 2020-07-21T03:58:59Z

This is a non-issue with WSL2

WSL2 is not a perfect fix. As described in the Best Practices section of the docs you must put your code inside the WSL VM if you want changes to the files to be detected. This makes working with your favorite IDE or other Windows tools more complicated. More info:

Updates of files outside the container are not visible inside of it #7276

[WSL2] Mounting linux path as volume not working #4812

microsoft/WSL#4739

microsoft/WSL#4064

True, it is not completely without issue, but it does resolve this one completely, which was our biggest pain point.

We have had far fewer issues with our dev ops process with WSL2. The VS Code integration is pretty great as well and the file sharing between Windows and WSL2 works well enough for the windows tools we use and has significantly sped up our development process.

Sawtaytoes · 2020-08-13T16:35:46Z

Before January 2020

In the past, watching for changes in mapped volumes required your apps polling for file changes from the Docker container. Many Node.js libraries came with a poll option to set a polling interval. This caused watching files to be very slow; almost unusable in Windows until January 2020 when native Windows file-watching was added.

WSL2

Now with WSL2, native file-watching is gone again. I have to poll files again to see changes. It's a huge step backward.

WSL2 is the recommended method of using Docker in Windows, but that doesn't make sense unless you're specifically using VS Code. Now, I could go back to polling like in the past, but then I'd opt out of using Docker altogether as that costs my projects time.

For everyone who doesn't use VS Code, if the Docker build is broken or you need to make changes before starting the container, that requires editing the project's files on your host OS's drive. But then when developing with hot-reload or some other file-watching technology, you'll need to start the project, setup your text editor to view project files from inside the Docker container, and then file-watching works again.

Is it on the roadmap for WSL2 Docker to allow for native Windows file-watching without having to edit files in the Docker container?

docker-robott · 2020-11-11T01:00:21Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

docker-robott · 2021-01-10T03:00:11Z

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

stephen-turner · 2021-02-25T14:58:19Z

@Sawtaytoes I only just saw your comment. In the case of WSL 2, Windows manages the file sharing from the Windows filesystem into the Linux environment, not us, so it is Microsoft who would have to add inotify support. That's one reason why we recommend keeping your files in the Linux filesystem: https://docs.docker.com/docker-for-windows/wsl/#best-practices.

Actually, I think they have some tickets already. https://github.com/microsoft/WSL/search?q=inotify&type=issues would be a good starting place.

simonferquel added the status/acknowledged label Jan 18, 2017

rn added the status/needs-information label Jan 19, 2017

simonferquel removed the status/acknowledged label Jan 19, 2017

dgageot added area/volumes status/acknowledged and removed status/needs-information labels Feb 2, 2017

docker-robott added the lifecycle/stale label Aug 19, 2018

docker-robott added the lifecycle/stale label Dec 10, 2019

docker-robott removed the lifecycle/stale label Dec 10, 2019

docker-robott added the lifecycle/stale label Apr 2, 2020

docker-robott removed the lifecycle/stale label Apr 2, 2020

docker-robott added the lifecycle/stale label Nov 11, 2020

docker-robott closed this as completed Dec 11, 2020

docker locked and limited conversation to collaborators Jan 10, 2021

docker-robott added the lifecycle/locked label Jan 10, 2021

Docker on Windows is not usable with volumes mapped to local directories #394

Docker on Windows is not usable with volumes mapped to local directories #394

Comments

denisvmedia commented Jan 17, 2017

Expected behavior

Actual behavior

Information

Steps to reproduce the behavior

Misc

simonferquel commented Jan 18, 2017

reypm commented Jan 20, 2017

denisvmedia commented Jan 21, 2017 • edited Loading

denisvmedia commented Jan 21, 2017

reypm commented Jan 21, 2017

denisvmedia commented Jan 21, 2017 • edited Loading

denisvmedia commented Jan 22, 2017

cfstras commented Mar 3, 2017

colinfindlay-nz commented Mar 23, 2017 • edited Loading

ezynda3 commented Mar 23, 2017

ovecka commented Apr 11, 2017

cfstras commented Apr 13, 2017 • edited Loading

vanko-dev commented Jun 8, 2017

YaronBasson commented Jul 2, 2017

kumarharsh commented Nov 10, 2017

philippslang commented Dec 4, 2017

andreikojusko commented Jan 11, 2018

Celoain commented Jan 29, 2018

dashesy commented May 8, 2018

denisvmedia commented May 20, 2018

dashesy commented May 21, 2018

docker-robott commented Aug 19, 2018

jpierront commented Aug 19, 2018

jonasschumacher commented Dec 6, 2018

kumarharsh commented Dec 10, 2018

technatelogy commented Dec 10, 2018

SC7639 commented Jan 16, 2019 • edited Loading

robertdumitrescu commented Feb 25, 2019

grexlort commented Mar 29, 2019

kumarharsh commented Mar 30, 2019 • edited Loading

chrisdfrey commented Apr 30, 2019 • edited Loading

zjcnew commented Sep 11, 2019

docker-robott commented Dec 10, 2019

sylvainar commented Dec 10, 2019

robertdumitrescu commented Jan 3, 2020

docker-robott commented Apr 2, 2020

nverwer commented Apr 2, 2020

Burgito commented Jun 11, 2020

technatelogy commented Jun 11, 2020

slominskir commented Jul 20, 2020

technatelogy commented Jul 21, 2020

Sawtaytoes commented Aug 13, 2020

Before January 2020

WSL2

docker-robott commented Nov 11, 2020

docker-robott commented Jan 10, 2021

stephen-turner commented Feb 25, 2021

denisvmedia commented Jan 21, 2017 •

edited

Loading

denisvmedia commented Jan 21, 2017 •

edited

Loading

colinfindlay-nz commented Mar 23, 2017 •

edited

Loading

cfstras commented Apr 13, 2017 •

edited

Loading

SC7639 commented Jan 16, 2019 •

edited

Loading

kumarharsh commented Mar 30, 2019 •

edited

Loading

chrisdfrey commented Apr 30, 2019 •

edited

Loading