Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not reconcile images and builds being deleted #1820

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

danail-branekov
Copy link
Contributor

When an image is being deleted, k8s does not delete the object immediately, instead it sets the deletion timestamp awaiting finalization. See
k8s docs for details

If the kpack image (or its builds) has a finalizer being added by an external component (such as
Korifi), then when the client request the object to be deleted, the image/build reconcilers keep reconciling the image/build causing new build pods to be created as the image is being deleted. Even though eventually the image gets deleted, it takes significant amount of time.

In order to address this, this PR "short-circuits" the image and build reconcilers to immediately return whenever an image or a build has their deletion timestamp set, thus preventing builds and build pods being rescheduled.

@codecov-commenter
Copy link

codecov-commenter commented Jan 22, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 13.33333% with 39 lines in your changes missing coverage. Please review.

Project coverage is 35.29%. Comparing base (49014e2) to head (e4ef85e).
Report is 101 commits behind head on main.

Files with missing lines Patch % Lines
pkg/reconciler/buildpack/buildpack.go 0.00% 6 Missing ⚠️
...kg/reconciler/clusterbuildpack/clusterbuildpack.go 0.00% 5 Missing ⚠️
pkg/reconciler/build/build.go 0.00% 4 Missing ⚠️
pkg/reconciler/builder/builder.go 0.00% 4 Missing ⚠️
pkg/reconciler/clusterbuilder/clusterbuilder.go 0.00% 4 Missing ⚠️
pkg/reconciler/clusterstack/clusterstack.go 0.00% 4 Missing ⚠️
pkg/reconciler/clusterstore/clusterstore.go 0.00% 4 Missing ⚠️
pkg/reconciler/image/image.go 0.00% 4 Missing ⚠️
pkg/reconciler/sourceresolver/sourceresolver.go 0.00% 4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1820       +/-   ##
===========================================
- Coverage   67.34%   35.29%   -32.05%     
===========================================
  Files         140      258      +118     
  Lines        8886    20001    +11115     
===========================================
+ Hits         5984     7060     +1076     
- Misses       2393    12339     +9946     
- Partials      509      602       +93     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@chenbh chenbh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels kinda awkward to have this short circuit on only the Build and Image, I wonder if we should just add this to all our reconcilers.

I imagine we can make use of the FilterFunc to skip any resource with non-nil DeletionTimestamps

imageInformer.Informer().AddEventHandler(controller.HandleAll(impl.Enqueue))
buildInformer.Informer().AddEventHandler(cache.FilteringResourceEventHandler{
FilterFunc: controller.FilterControllerGK(buildapi.SchemeGroupVersion.WithKind(Kind).GroupKind()),
Handler: controller.HandleAll(impl.EnqueueControllerOf),
})

Something like pkg/reconciler/filter.go:

func FilterDeletionTimestamp(obj interface{}) bool {
	object, ok := obj.(metav1.Object)
	if !ok {
		return false
	}

	return object.GetDeletionTimestamp() == nil
}

And in each reconciler:

- informer.Informer().AddEventHandler(controller.HandleAll(impl.Enqueue))

+ informer.Informer().AddEventHandler(cache.FilteringResourceEventHandler{ 
+  	FilterFunc: reconciler.FilterDeletionTimestamp, 
+  	Handler:    controller.HandleAll(impl.Enqueue), 
+  }) 

This way we can also get away without modifying all the reconciler tests

danail-branekov and others added 3 commits February 24, 2025 16:01
When an image is being deleted, k8s does not delete the object
immediately, instead it sets the deletion timestamp awaiting
finalization. See [k8s
docs](https://github.com/kubernetes/apimachinery/blob/45d29dc4d66fc2ac83e736e79752ad81a9c6195f/pkg/apis/meta/v1/types.go#L190-L209)
for details

If the kpack image (or its builds) has a finalizer being added by an
external component (such as
[Korifi](https://github.com/cloudfoundry/korifi/blob/17557eb68fed830f3f57abd651882a712fc25f5f/kpack-image-builder/controllers/webhooks/finalizer/finalizer_webhook.go#L25)),
then when the client request the object to be deleted, the image/build
reconcilers keep reconciling the image/build causing new build pods to
be created as the image is being deleted. Even though eventually the
image gets deleted, it takes significant amount of time.

Signed-off-by: Danail Branekov <danailster@gmail.com>
Co-authored-by: Georgi Sabev <georgethebeatle@gmail.com>
This reverts commit 1d6a5b2.

Co-authored-by: Danail Branekov <danailster@gmail.com>
If the kpack image (or its builds) has a finalizer being added by an
external component, then when the client request the object to be
deleted, the image/build reconcilers keep reconciling the image/build
causing new build pods to be created as the image is being deleted. Even
though eventually the image gets deleted, it takes significant amount of
time.

Co-authored-by: Georgi Sabev <georgethebeatle@gmail.com>
@georgethebeatle georgethebeatle force-pushed the skip-reconciling-deleted-images branch from 5ff9f2a to e4ef85e Compare February 25, 2025 12:54
@georgethebeatle
Copy link
Contributor

@chenbh thanks for the suggestion. We just pushed a new commit, is that what you meant?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants