[BUG] Overlay network not found on worker node (2.33.0/v28.0.0 Again/Still)

[Fabbzz]

Description

As described in bug #11894 there are still issues with local containers attaching themselves to an attachable overlay network.

I just upgraded to docker v28.0.0 where this went fine once after the upgrade; but there is more:

It seems that it's prior to some networks and some not; the question remains why. It looks like it happens when you have too much of these types of connecting networks/containers and then bulksync (on one of the nodes) is maxing out in time like 30 sec. Some hosts/networks is what I can re-compose all day long without any issue.

I have been testing with removals, sleep times, --force whatever I can but it still seems to happen.

The docker run solutions with a pseudo/zombie container being only attached still fixes this issue but it's a no-go at the end.

I hope we can get this fixed as soon as possible in some default way

Steps To Reproduce

See #11894

Compose Version

2.33.0

Docker Environment

Client: Docker Engine - Community
 Version:    28.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 9
  Running: 5
  Paused: 0
  Stopped: 4
 Images: 7
 Server Version: 28.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: jq1ajzsofbowgcz7lxktcs1tq
  Is Manager: true
  ClusterID: 4w6riymm34xdjor7546t7tzsn
  Managers: 3
  Nodes: 3
  Default Address Pool: 10.1.1.0/24  
  SubnetSize: 28
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 0
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.0.1.11
  Manager Addresses:
   10.0.1.11:2377
   10.0.1.12:2377
   10.0.1.13:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-53-generic
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.73GiB
 Name: docker-host-01.domain.tld
 ID: d7238096-908d-4b87-9a78-2765fb0ce919
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 111
  Goroutines: 286
  System Time: 2025-02-21T10:36:03.993012621+01:00
  EventsListeners: 3
 Experimental: true
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.0.0.0/24, Size: 28

Anything else?

Let's don't give up on this; we can fix it!

[ndeloof]

Can you please try to reproduce using DinD as described on #11894 (comment) ?

Without a reproduction example I can't give you additional assistance :'(

[Fabbzz]

@ndeloof Sure I want to but what does it matter as it doesn't work by default ?

I used DinD before but this happens on production so I need to see how I can test there.

[maxens-git]

same

[ez839]

#Fix