MCO-643: MCO-645: Reintroduce MCS CA rotation

[djoshy]

This PR re-applies the work done in #4669, which was reverted in #4713. We should only land this after openshift/installer#9309

How to verify:
In addition to passing existing e2es and verification done in #4669 , agent installations should succeed. The recently added test in the MCO repo which tests agent workflows(ci/prow/e2e-agent-compact-ipv4) should give us a good signal on this.

[openshift-ci-robot]

@djoshy: This pull request references MCO-1457 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

This PR does the following:

• Re-applies the work done in MCO-643: MCO-645: Implement a rotation mechanism for the MCS TLS cert #4669, which was revert in OCPBUGS-44832: Revert MCS CA rotation #4713
• Implements generation of the MCS CA/cert during bootstrap, this should help

How to verify:
In addition to passing existing e2es and verification done in #4669 , agent installations should succeed. The recently added test in the MCO repo which tests agent workflows(ci/prow/e2e-agent-compact-ipv4) should give us a good signal on this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@djoshy: This pull request references MCO-1457 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

This PR re-applies the work done in #4669, which was reverted in #4713. We should only land this after openshift/installer#9309

How to verify:
In addition to passing existing e2es and verification done in #4669 , agent installations should succeed. The recently added test in the MCO repo which tests agent workflows(ci/prow/e2e-agent-compact-ipv4) should give us a good signal on this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/testwith openshift/machine-config-operator/master/e2e-agent-compact-ipv4 #4735 openshift/installer#9309

[openshift-ci-robot]

@djoshy: This pull request references MCO-643 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

This PR re-applies the work done in #4669, which was reverted in #4713. We should only land this after openshift/installer#9309

How to verify:
In addition to passing existing e2es and verification done in #4669 , agent installations should succeed. The recently added test in the MCO repo which tests agent workflows(ci/prow/e2e-agent-compact-ipv4) should give us a good signal on this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

openshift/installer#9309 has landed, so we should be good to go ahead with this.

[djoshy]

/test unit

[djoshy]

/retest-required

[yuqi-zhang]

/lgtm
/hold

This should be good to go, the unrevert looks clean. Holding for a quick QE review just in case as we discussed, but only for the new behaviour now that the installer PR has merged so we aren't doing unnecessary rotations on installs.

[yuqi-zhang]

Just curious, was the client-go bump needed for a test or something?

[djoshy]

Yeah, it's needed for a fake machine client for one of the units. I'm not sure when/how, but our current client-go vendor was missing them. I only semi-bumped it to 20241203091221 instead of the latest because it would bring in the OCL changes and that breaks a few devex units.

[openshift-ci]

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-op-ocl	9117787	link	false	/test e2e-gcp-op-ocl
ci/prow/e2e-vsphere-ovn-upi	9117787	link	false	/test e2e-vsphere-ovn-upi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ptalgulk01]

Pre-merge verification:
Steps To verify: https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-77784

• Verified using IPI based AWS, Baremetal cluster
• For verifying on agent based Baremetal cluster using Cluster-Bot cmd

workflow-launch baremetal-lab-agent-install-pxe [registry.build05.ci.openshift.org/ci-ln-63i7b1b/release:latest](http://registry.build05.ci.openshift.org/ci-ln-63i7b1b/release:latest) "AUX_HOST=[openshift-qe-metal-ci.arm.eng.rdu2.redhat.com](http://openshift-qe-metal-ci.arm.eng.rdu2.redhat.com/)","architecture=amd64","DISCONNECTED=false","RESERVE_BOOTSTRAP=false","AGENT_PLATFORM_TYPE=none","ipv4_enabled=true","ipv6_enabled=false"

• For non-machinesets based cluster certificate are not rotated

oc logs machine-config-controller-5c45b49479-5xw2d | grep -i cert
Defaulted container "machine-config-controller" out of: machine-config-controller, kube-rbac-proxy
I0116 05:48:05.614698       1 certrotation_controller.go:169] MCS CA/TLS cert rotator not added
....
I0116 05:48:05.621896       1 certrotation_controller.go:188] Starting machineconfigcontroller-certrotationcontroller
I0116 05:48:05.621907       1 certrotation_controller.go:176] Waiting for machineconfigcontroller-certrotationcontroller
I0116 05:48:05.722068       1 certrotation_controller.go:184] Finished waiting for machineconfigcontroller-certrotationcontroller
I0116 05:48:05.722073       1 certrotation_controller.go:194] No cert rotators needed, shutting down
I0116 05:48:05.722076       1 certrotation_controller.go:195] Shutting down machineconfigcontroller-certrotationcontroller

For agent based with machinesets cluster will verify after PR is merged.

Adding qe-approved label

/label qe-approved

[openshift-ci-robot]

@djoshy: This pull request references MCO-643 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

This PR re-applies the work done in #4669, which was reverted in #4713. We should only land this after openshift/installer#9309

How to verify:
In addition to passing existing e2es and verification done in #4669 , agent installations should succeed. The recently added test in the MCO repo which tests agent workflows(ci/prow/e2e-agent-compact-ipv4) should give us a good signal on this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/payload-job
periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-compact-fips

/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance

/payload-job
periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-single-node-ipv6

[openshift-ci]

@djoshy: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-compact-fips
• periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance
• periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-single-node-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/18bc7240-d8b8-11ef-8f3c-f26c3537d1f7-0

[djoshy]

/payload 4.19 nightly blocking

[openshift-ci]

@djoshy: trigger 14 job(s) of type blocking for the nightly release of OCP 4.19

• periodic-ci-openshift-release-master-ci-4.19-e2e-aws-upgrade-ovn-single-node
• periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-upgrade
• periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-upgrade
• periodic-ci-openshift-release-master-ci-4.19-upgrade-from-stable-4.18-e2e-gcp-ovn-rt-upgrade
• periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-ovn-conformance
• periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-serial
• periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview
• periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial
• periodic-ci-openshift-release-master-nightly-4.19-fips-payload-scan
• periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm
• periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-ipv6
• periodic-ci-openshift-microshift-release-4.19-periodics-e2e-aws-ovn-ocp-conformance
• periodic-ci-openshift-microshift-release-4.19-periodics-e2e-aws-ovn-ocp-conformance-serial
• periodic-ci-openshift-release-master-nightly-4.19-e2e-rosa-sts-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0bcee5c0-d8ba-11ef-8aae-230e8c73c24b-0

[djoshy]

/retest-required

[djoshy]

/label acknowledge-critical-fixes-only

/unhold

[yuqi-zhang]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[djoshy]

/hold

[djoshy]

/unhold

[djoshy]

/hold

[djoshy]

/unhold

[djoshy]

/test e2e-agent-compact-ipv4

[djoshy]

/test unit

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 9871a24 and 2 for PR HEAD 9117787 in total

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.19.0-202501242239.p0.g38fb1b8.assembly.stream.el9.
All builds following this will include this PR.

[ptalgulk01]

Post-merge verification -
Fix included in built registry.ci.openshift.org/ocp/release:4.19.0-0.nightly-2025-01-25-021606
The below periodic agent jobs has not observed error related to MCS TLS cert

• https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-4.19-upgrade-from-stable-4.18-vsphere-agent-sno-f28/1883807409026109440
• https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-baremetal-pxe-compact-agent-ipv4-static-connected-f14/1883806520756408320
• https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-baremetal-pxe-sno-agent-ipv4-static-connected-f14/1883806129645948928