Update drain lib to support unreachable nodes

[michaelgugino]

User Story

As a operator I would like to successfully drain unreachable nodes when deleting them to ensure PDBs are respected.

Detailed Description

Currently, if an unreachable Node has pods with local storage, if those pods pass PDB checks (eviction succeeds), they will get a deletion timestamp but will never be deleted. We should account for this by ignoring them after X time by upgrading the drain lib to kubernetes/kubectl@846b394 or newer and implementing the new option "SkipWaitForDeleteTimeoutSeconds" when the node is unreachable. If the node is ready, do not set this option or set it to 0 to disable the behavior.

Time could be configurable, I suggest 5 minutes as a starting point for now.

Anything else you would like to add:

Steps to replicate current problem:

Create a pod with local storage.
Disable the kubelet daemon/process on that particular node.
Attempt to evict/delete the pod.
Pod never goes away.

/kind feature

edit: s/unready/unreachable/

[hypnoglow]

/assign

[enxebre]

and implementing the new option "SkipWaitForDeleteTimeoutSeconds" when the node is unready

shouldn't this rather be when node is unreachable?

[michaelgugino]

and implementing the new option "SkipWaitForDeleteTimeoutSeconds" when the node is unready

shouldn't this rather be when node is unreachable?

Yes, I think it probably should be unreachable. Unready might mean the node is able to schedule/remove pods normally, but unreachable implies the kubelet is gone/crashed/etc.