Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear resource version before updating management components version #7787

Conversation

cxbrowne1207
Copy link
Member

@cxbrowne1207 cxbrowne1207 commented Mar 5, 2024

Issue #, if available:

Description of changes:
After the CLI is upgrading the core components, we update the management components version annotation on the Cluster object using a server-side apply. Despite retrieving the most recent version of the object before performing the update, there can be a case where the object has been updated by another process in the meantime (e.g. the controller). This creates conflict, the update will fail and a retry is suggested

This PR clears the resource version of the Cluster object before applying. This avoids triggering the api server to detect a conflict if the resource versions are different.

Error message

Error: failed to upgrade cluster: applying object with kubectl: error: Operation cannot be fulfilled on clusters.anywhere.eks.amazonaws.com "main-i-0b9f7-011f2db": the object has been modified; please apply your changes to the latest version and try again
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

Testing (if applicable):

  • Unit tests
    • The existing unit tests for upgrading the core components cover the rest.
  • Manually ran the docker upgrade from latest minor release test
    • Paused it just before the server side apply operation with a time.Sleep
    • Edited the Cluster object using kubectl so that the .metadata.resourceVersion is updated.
    • Observed the error before the fix and after the fix.
--- PASS: TestDockerKubernetes128to129UpgradeFromLatestMinorRelease (1174.09s)
PASS

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@eks-distro-bot
Copy link
Collaborator

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@eks-distro-bot eks-distro-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 5, 2024
@cxbrowne1207 cxbrowne1207 force-pushed the retry-update-management-components-version branch from cca1f2c to 3e39cff Compare March 5, 2024 21:49
@cxbrowne1207 cxbrowne1207 marked this pull request as ready for review March 5, 2024 22:05
@cxbrowne1207 cxbrowne1207 changed the title Add retrier when updating management components version Add retry when updating management components version Mar 5, 2024
@cxbrowne1207 cxbrowne1207 force-pushed the retry-update-management-components-version branch from 3e39cff to 82d5d26 Compare March 5, 2024 23:15
@cxbrowne1207 cxbrowne1207 changed the title Add retry when updating management components version Clear resource version before updating management components version Mar 5, 2024
@cxbrowne1207
Copy link
Member Author

cxbrowne1207 commented Mar 5, 2024

/hold

Running the test again

@cxbrowne1207 cxbrowne1207 force-pushed the retry-update-management-components-version branch from 82d5d26 to 0722b8c Compare March 5, 2024 23:38
@cxbrowne1207
Copy link
Member Author

/unhold

@cxbrowne1207 cxbrowne1207 force-pushed the retry-update-management-components-version branch from 0722b8c to 63318d8 Compare March 5, 2024 23:39
@cxbrowne1207
Copy link
Member Author

/hold

Working of addressing unit test failures

Copy link

codecov bot commented Mar 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.61%. Comparing base (4583834) to head (0db8649).
Report is 179 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7787      +/-   ##
==========================================
+ Coverage   73.48%   73.61%   +0.13%     
==========================================
  Files         579      588       +9     
  Lines       36357    37175     +818     
==========================================
+ Hits        26718    27368     +650     
- Misses       7875     8015     +140     
- Partials     1764     1792      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cxbrowne1207
Copy link
Member Author

/unhold

@cxbrowne1207
Copy link
Member Author

/unhold

@tatlat
Copy link
Member

tatlat commented Mar 6, 2024

/lgtm

@cxbrowne1207
Copy link
Member Author

/approve

@eks-distro-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cxbrowne1207

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eks-distro-bot eks-distro-bot merged commit b6e9ad5 into aws:main Mar 6, 2024
10 checks passed
@cxbrowne1207 cxbrowne1207 deleted the retry-update-management-components-version branch March 6, 2024 21:27
@cxbrowne1207
Copy link
Member Author

/cherry-pick release-0.19

@eks-distro-pr-bot
Copy link
Contributor

@cxbrowne1207: new pull request created: #7799

In response to this:

/cherry-pick release-0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants