systemd-watchdog: improve notifications to watchdog

[alexmohr]

systemd will now be notified on send timeouts
and when event handling is running for a long time

this prevents dlt-daemon from being killed by
systemd watchdog when processing log messages
would take longer than the watchdog timeout

To test we need the following things

• dlt-daemon started via systemd with enabled watchdog
• a lot of logging going into the daemon

Excecute the script below on your test system.
It spawns a lot of dlt-receive proceses, the goal is for dlt-daemon to survive.

end_time=$((SECONDS+180))

while [ $SECONDS -lt $end_time ]; do 
  dlt-receive -a 127.1 &
done 

killall -9 dlt-receive

if [[ $(uname) == "Linux" ]]; then
  systemctl status dlt.service
  echo Service should be running for > 3 minutes
else
  pidin -faAt | grep dlt-daemon | grep -v grep
  echo Service should be running at least since system boot
fi

The program was tested solely for our own use cases, which might differ from yours.
Licensed under Mozilla Public License Version 2.0

^{Alexander Mohr, alexander.m.mohr@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH, imprint}

[michael-methner]

Hello @alexmohr ,
thanks for the patch. It looks good but I haven't touched the systemd_notify code yet.

One thing which I wonder about is that the watchdog is triggered at several places instead from one central place. E.g. in case the poll in dlt_daemon_handle_event() returns. Would this make more sense?

[alexmohr]

Hi @michael-methner,

it's triggered from multiple places to prevent watchdog timeouts when we run into socket timeouts.
Especially if many (or multiple slow) clients are connected it can happen that the dlt-daemon is running into multiple socket timeouts back to back.
If we would not trigger the systemd watchdog here we might run into a systemd watchdog timeout although the daemon is still working properly.
Triggering from the event handler alone is therefore not sufficient because the watchdog timeout could be over when we are back in the event handler function

[michael-methner]

Is there a reason why dlt_daemon_trigger_systemd_watchdog_if_necessary() was not used here?

[alexmohr]

Because there is no timestamp to use as a reference time. We could call the function and always pass 0 as timestamp.
As you mentioned in the other comment this patch improves the watchdog behavior but it's far from perfect.
Especially in environment with congested networks between the clients and the daemon combined with heavy logging load.

[michael-methner]

Got it. Thanks.

[michael-methner]

Hi @michael-methner,

it's triggered from multiple places to prevent watchdog timeouts when we run into socket timeouts. Especially if many (or multiple slow) clients are connected it can happen that the dlt-daemon is running into multiple socket timeouts back to back. If we would not trigger the systemd watchdog here we might run into a systemd watchdog timeout although the daemon is still working properly. Triggering from the event handler alone is therefore not sufficient because the watchdog timeout could be over when we are back in the event handler function

Thanks for the explanation. As the change is definitely an improvement, I will merge it. But that slow clients may still cause watchdog timeouts worries me. But I understand that this will be part of a bigger rework.