Resolved Service Annotation Loop Issue in ROSA Environment #739
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
March 20, 2024 09:15
In ROSA (RedHat OpenShift on AWS), we noticed that when setting up a network LoadBalancer an annotation would automatically be added to the Service object. This caused the operator to start a reconcile loop. It would remove the annotation, only to have OpenShift add it back. So, the operator was in a continuous reconcile loop. To fix this we now allow manual annotations be added to service objects. The operator will only ensure the annotations that it generates are the correct value. It will ignore any additional annotation that was added outside of the VerticaDB. I am also cleaning up the PVC expansion events. This can cause quite a lot of noise about skipping expansion if we continuously are doing reconciles. The skip events have been changed to log entries instead.
roypaulin
approved these changes
Mar 20, 2024
| apiVersion: vertica.com/v1 | ||
| kind: VerticaDB | ||
| name: v-pvc-expansion | ||
| # No event to check if expansion is skipped |
There was a problem hiding this comment.
Why do you still keep this file?
There was a problem hiding this comment.
If we run with a PVC that allows for expansion, then the step can still be used to verify that.
|
Is this related to the crash from yesterday? |
Yes, we were repeatedly doing reconcile iterations. Eventually, the K8s OOMKiller stepped in and killed the pod. We were allocating/freeing too much memory that the garbage collection couldn't keep up. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In ROSA (RedHat OpenShift on AWS), we noticed that when setting up a network LoadBalancer an annotation would automatically be added to the Service object. This caused the operator to start a reconcile loop. It would remove the annotation, only to have OpenShift add it back. So, the operator was in a continuous reconcile loop.
To fix this we now allow manual annotations be added to service objects. The operator will only ensure the annotations that it generates are the correct value. It will ignore any additional annotation that was added outside of the VerticaDB.
I am also cleaning up the PVC expansion events. This can cause quite a lot of noise about skipping expansion if we continuously are doing reconciles. The skip events have been changed to log entries instead.