MachineOf should support wildcards

[tomprince]

No description provided.

[bcwaldon]

What's the use-case here? MachineOf is only intended to match a single unit, while a wildcard implies multiple matches.

[graemej]

We have a usecase where I think wildcards would help.

We have a container, let's call it 'bedrock', that provides shared services (Splunk logging, statsd monitoring) and want to run one copy per host. We can use X-Conflicts and a pattern that matches our container names (e.g. bedrock@1.service, bedrock@2.service) to achieve this. Provided the number of bedrock instances >= cluster size we get the behaviour we're after with a bedrock@<instance>.service like this:

[Unit]
Description=Shared Services
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/etc/environment
ExecStart=/usr/bin/docker run <container>

[Install]
WantedBy=multi-user.target

[X-Fleet]
X-Conflicts=bedrock@*.service

Wildcards would allow us to define unit that depend on bedrock services without tying it to a specific instance/machine. We really don't care about which bedrock instance an application uses as long as the service is there. Giving Fleet more flexibility about where it places work feels appealing.

I can see wanting wildcards in both the [X-Fleet] and [Unit] sections so we can get the dependencies right, in a myapp.service like this:

[Unit]
Description=MyApp
After=docker.service bedrock@*.service
Requires=docker.service bedrock@*.service

[Service]
ExecStart=/usr/bin/docker run <myapp>

[Install]
WantedBy=multi-user.target

[X-Fleet]
MachineOf=bedrock@*.service

[bcwaldon]

@graemej I've been thinking of the wildcard as a bash expansion. For example, you would effectively have MachineOf=bedrock@1.service bedrock@2.service bedrock@3.service ... bedrock@n.service which implies multiple required conditions that must be met. Are you thinking of this wildcard as matching any other unit that fulfills the pattern?

[graemej]

I was thinking 'any-of' vs. 'all-of' semantics on the match. e.g. have a dependent unit scheduled anywhere that bedrock is running.

[oddurmagg]

I am looking into this as well to distribute units that take care of shared utilitites(just like in your case, log-forwarding, monitoring etc).

Would it not be possible to use a metadata tag for this ? Tagging all the machines which should get the utility containers with some tag, say

and then having

[X-Fleet]
MachineMetadata="hasSharedContainers=yes"

Would that not map the unit to all the machines which have that tag ?

[oddurmagg]

After reading the docs further, it seems a unit will only ever be assigned to a single machine.

I guess I could automatically start the utility containers on all machines via the cloud-config for those machines as shown here: http://coreos.com/docs/cluster-management/setup/cloudinit-cloud-config/

[bcwaldon]

@oddurmagg We've been considering supporting a static schedule of units that a fleet Agent must start and track during initialization. It's effectively the same as the coreos.units section in our cloud-config, but with a couple of major benefits:

1. fleet could factor the resources consumed by these statically-scheduled units into scheduling decisions
2. the units could benefit from any debugging/monitoring/logging functionality fleet provides in the future

[graemej]

We considered the cloud-config path too (we're PXE-booting bare metal) but fleet seemed cleaner. The sticking point is that we want units that depend on a service container to have the right systemd dependencies (i.e. Requisite=) so they come down tool if the service container fails for some reason.

Although CoreOS is fast to boot it felt like rolling out service containers via fleet would be smoother if an upgrade was needed.

[oddurmagg]

@bcwaldon I think having a way to say, "schedule this unit on all machines which have this metadata tag" would solve this. My concern is if I would configure, say aws autoscaling, to introduce new machines with a webapp on it, I would have to make sure there was a unit waiting to be scheduled on it. The autoscaling would then have to submit that unit start it so it would get scheduled on the newly introduced host.

I´d rather that machine would spin up, fleet would startup on it and from it´s metadata it would figure out what should be running on it.

It might not jive with fleet models, which might be assuming that each unit only runs on a single machine.

[bcwaldon]

So I think there are a few reasonable approaches here:

1. Place a static manifest of units in a cloud-config. Each fleet agent would start its units locally and report their state through the normal means. These units would not migrate on loss of that machine.
2. Place a static manifest of units in fleet's registry (etcd). Each fleet agent watches this central repository of units, cloning and starting units locally similar to option 1 above.
3. Build a small service that uses fleet's API to track the current machines, (un)scheduling a set of units on machine join/loss.

[oddurmagg]

1. would bake the units into the cloud-config which smells bad to me. I would want me service orchestration and configuration not to live there.
2. sounds like what I would want. Additionally being able to filter on machine metadata, which could be used to assign units to machines based on roles( which is basically what I want to achieve )
3. this sounds like building a service orchestrator on top of a service orchestrator.

[litch]

I'd also really like this. I'm using the ambassador pattern described here and have a unit that needs to be able to communicate with one service, so I'd like to run it on a box with <service>-dyn-amb@*.service, rather than tying it to any infrastructure or cloud-config-defined services.

[pikeas]

How about adding MachineAll with wildcard support?

[jonboulle]

@oddurmagg , I think the use cases you've described in this ticket should now be adequately covered by #681 and #811 - please follow along there.

[digital-wonderland]

I think I have a valid use case for this: I'm using an Elasticsearch cluster running on some hosts in a small CoreOS cluster to store data collected with Logstash. Logstash creates a new index every day. To do some housekeeping (optimize any index older than a day since it will be read from only and delete any index older than 30 days) I rely on elasticsearch curator.

For this curator instance it doesn't matter where it runs, as long as it runs on a host where Elasticsearch itself is running since it will be available at ${COREOS_PRIVATE_IPV4}:9200. Therefore I would like to schedule the curator instance with something like the following: MachineOf=elasticsearch@*.service.

Reading the above discussion about any of vs all of I think both points are valid. A solution might be to have MachineOf with all of semantics and MachineAnyOf with any of semantics?

[nrandell]

I've got a similar issue. I have one service that is shared between multiple clients. Ideally I want to be able to launch any number of these shared instances and have fleet run them where it decides to, and then have a number of clients connecting to the local instance.
I've worked around this by having a bash script pre-process the service files to produce a number of service files, with each file having a hardcoded 'MachineOf' entry in.

e.g. service@0.service -> MachineOf=shared@0.service
service@1.service -> MachineOf=shared@1.service
service@2.service -> MachineOf=shared@0.service
service@3.service -> MachineOf=shared@1.service

Some questions people may have.
Why can't each service have a single shared service it uses? The shared service is quite memory hungry and I don't want multiple instances taking up memory

Why can't I just have one shared service? I'm running on restricted resource machines and only have 2 CPUs per machine and want to be able to shard the shared service.

[devurandom]

I have this generate-serviceaccount-key.service unit:

[Unit]
Description=Generate service-account key file

[Service]
ExecStartPre=-/usr/bin/mkdir -p /opt/share/kube
ExecStart=/bin/openssl genrsa -out /opt/share/kube/kube-serviceaccount.key 2048 2>/dev/null
RemainAfterExit=yes
Type=oneshot

[X-Fleet]
MachineOf=kube-apiserver@*.service

kube-apiserver@.service depends on it:

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=etcd2.service generate-serviceaccount-key.service
After=etcd2.service generate-serviceaccount-key.service

[Service]
…

[X-Fleet]
Global=true
MachineMetadata=k8srole=master

The reason I need the wildcard is that I need generate-serviceaccount-key.service on every node running any instance of kube-apiserver@.service.

What currently happens is that kube-apiserver@${foo}.service is started, but generate-serviceaccount-key.service is not. Not on any machine of the cluster, at all. At the very least there should be some error indicating that the dependencies cannot be satisfied…

[LukasRos]

I have a similar scenario like @graemej and @digital-wonderland where I'm running multiple instances of one container and want a second container to be scheduled on any machine that runs the first container.

The first container is redis@.service which I don't want to make global, so I'm running e.g. redis@1.service and redis@2.service. The second container, application-x.service, needs only one instance but it relies on a local Redis server available.

Currently I'm required to set e.g. MachineOf=redis@1.service, effectively binding it to a specific other unit, whereas MachineOf=redis*.service would make more sense because it does not matter whether the application accesses redis@1.service or redis@2.service. Support for this would be really helpful.

[Dynom]

Apologies for the duplicate feature request, but I've outlined my use-case in #1298. I currently work around this feature by defining identical unit-files with a different name since I see no way to solve it any other way.

[christ0pher]

+1 for this feature

[wuqixuan]

+1 for this feature