nginx is aware of hosts file changes

[dark5un]

Why

Looking at this issue, I feel the solution can be fairly easy. monit can track file changes and can have nginx process react to these changes. Sending a SIGHUP signal to the parent nginx process is enough for the workers to refresh.

How to test

run touch /etc/hosts and see that nginx workers get replaced:

# ps fax | grep nginx
  47130 ?        S<     0:00 nginx: master process nginx -p /var/vcap/sys/tmp/nginx -c /var/vcap/jobs/nginx/config/nginx.conf
  47441 ?        S<     0:00  \_ nginx: worker process
  47442 ?        S<     0:00  \_ nginx: worker process
....
# ps fax | grep nginx
  47130 ?        S<     0:00 nginx: master process nginx -p /var/vcap/sys/tmp/nginx -c /var/vcap/jobs/nginx/config/nginx.conf
  47556 ?        S<     0:00  \_ nginx: worker process
  47557 ?        S<     0:00  \_ nginx: worker process

[benjaminguttmann-avtq]

@dark5un I am wondering, when I get the issue right, nginx does not get any information about the change. So I would not expect that any file in nginx changes because only the other VMs, e.g. grafana are touched by Bosh in that case, or am I wrong?

[dark5un]

@benjaminguttmann-avtq thank you for taking the time to look into this.

In our case, the grafana address as rendered here (https://github.com/bosh-prometheus/prometheus-boshrelease/blob/master/jobs/nginx/templates/config/nginx.conf#L175) is a bosh dns provided fqdn.

Nginx resolves that fqdn on startup and reload only. The ip address is stored in memory at this time.

If during a bosh deployment the ip for the fqdn changes, the bosh-agent will change the hosts file on the node, however it will not trigger an nginx restart as the config for nginx has not been altered.

[benjaminguttmann-avtq]

I think it will not harm to have that and maybe improve the situation :) Thx @dark5un

[dark5un]

Thank you for allowing this @benjaminguttmann-avtq . It will save us for sure.