[helm] add the ability to specify hostIP of karmada apiserver

[zhzhuang-zju]

What type of PR is this?
/kind feature

What this PR does / why we need it:
When installing Karmada using Helm, if we want to access the karmada-apiserver externally, the following issues will arise:

• The format of the clusters server in the generated kubeconfig is: https://{{ $name }}-apiserver.{{ $namespace }}.svc.{{.Values.clusterDomain }}:5443, which cannot be directly accessed externally.
• The format of the server provided by the generated cluster-info ConfigMap is https://{{ $name }}-apiserver.{{ $namespace }}.svc.{{.Values.clusterDomain }}:5443, which cannot be directly accessed externally.
• If we manually replace the above server IP with the host IP of the karmada-apiserver, when accessing the karmada-apiserver, the certificate verification will fail because the host IP is not in the Subject Alternative Name list of the certificate.

In summary, these issues will be addressed by adding the ability to specify the host IP of the karmada-apiserver.

Which issue(s) this PR fixes:
Fixes #3594

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`helm`: add the ability to specify hostIP of karmada-apiserver

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chaosi-zju for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• charts/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zhzhuang-zju]

cc @chaosi-zju @yyzxw

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.09%. Comparing base (d80b7d4) to head (50377ae).
Report is 4 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6097      +/-   ##
==========================================
- Coverage   48.11%   48.09%   -0.03%     
==========================================
  Files         668      668              
  Lines       55163    55211      +48     
==========================================
+ Hits        26544    26551       +7     
- Misses      26893    26933      +40     
- Partials     1726     1727       +1     

Flag	Coverage Δ
unittests	48.09% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chaosi-zju]

what about directly set hostNetwork=true as default

• if hostIP is specified, use hostIP as karmada-apiserver host ip
• if not specified, fetch node ip from kube-apiserver as karmada-apiserver host ip (like kubectl get ep kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}')

[zhzhuang-zju]

what about directly set hostNetwork=true as default

When hostNetwork is false, the hostIP can also be combined with nodePort to form a server for use.

if not specified, fetch node ip from kube-apiserver as karmada-apiserver host ip (like kubectl get ep kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}')

Good idea. What I'm concerned about is whether the host IPs of the kube - apiserver and the karmada - apiserver must be the same in a multi-node environment.

[chaosi-zju]

Good idea. What I'm concerned about is whether the host IPs of the kube - apiserver and the karmada - apiserver must be the same in a multi-node environment.

Make sense, as for multi-node k8s, it cann't be limited to considering the current node IP, we should add all nodeIP to certificate SANs.

BTW, based on your considerations, let's deduce the user's behavior, if I want to specify the .Values.apiServer.hostIP, how can I predict what the hostIP will be before installing karmada in a multi-node cluster?

also, once the pod/karmada-apiserver migrate to another node, how to deal with the problem that the IP of the new node is not in SANs?

[chaosi-zju]

simlar issue proposed two years ago: #4013

[zhzhuang-zju]

also, once the pod/karmada-apiserver migrate to another node, how to deal with the problem that the IP of the new node is not in SANs?

This is a good question. In this case, if you want to migrate the pod/karmada-apiserver without affecting external access to the karmada-apiserver, you can use a Service of type NodePort, and fill in the hostIP with the IP address of any node.

helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --set apiServer.hostNetwork=false,apiServer.hostIP=${HOST_IP},apiServer.serviceType=NodePort,apiServer.nodePort=31333

[chaosi-zju]

you can use a Service of type NodePort, and fill in the hostIP with the IP address of any node

so, then how can I access apiserver? by node ip?

[zhzhuang-zju]

so, then how can I access apiserver? by node IP?

By any node ip

[zhzhuang-zju]

so, then how can I access apiserver? by node ip?

1. create a multi-node cluster

$ kubectl get nodes -owide
member4-control-plane    Ready    control-plane   69s   v1.31.2   172.18.0.8    <none>        Debian GNU/Linux 12 (bookworm)   4.15.0-213-generic   containerd://1.7.18
member4-control-plane2   Ready    control-plane   58s   v1.31.2   172.18.0.9    <none>        Debian GNU/Linux 12 (bookworm)   4.15.0-213-generic   containerd://1.7.18
member4-control-plane3   Ready    control-plane   49s   v1.31.2   172.18.0.7    <none>        Debian GNU/Linux 12 (bookworm)   4.15.0-213-generic   containerd://1.7.18
member4-worker           Ready    <none>          38s   v1.31.2   172.18.0.10   <none>        Debian GNU/Linux 12 (bookworm)   4.15.0-213-generic   containerd://1.7.18
member4-worker2          Ready    <none>          38s   v1.31.2   172.18.0.6    <none>        Debian GNU/Linux 12 (bookworm)   4.15.0-213-generic   containerd://1.7.18

2. install karmada by helm

$ helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --set apiServer.hostNetwork=false,apiServer.hostIP=172.18.0.7,apiServer.serviceType=NodePort,apiServer.nodePort=31333
$ kubectl get pods --namespace karmada-system -owide
NAME                                               READY   STATUS              RESTARTS   AGE   IP          NODE              NOMINATED NODE   READINESS GATES
etcd-0                                             0/1     ContainerCreating   0          30s   <none>      member4-worker    <none>           <none>
karmada-aggregated-apiserver-6d7fd4cbf7-9knhh      0/1     Init:0/1            0          30s   <none>      member4-worker2   <none>           <none>
karmada-apiserver-64d6ccf69b-lnzhg                 0/1     Init:0/1            0          30s   10.12.6.4   member4-worker    <none>           <none>
karmada-controller-manager-5f8c78b849-k55sq        0/1     Init:0/1            0          30s   10.12.6.5   member4-worker    <none>           <none>
karmada-kube-controller-manager-7b866cc57c-hcq6c   0/1     Init:0/1            0          30s   <none>      member4-worker2   <none>           <none>
karmada-scheduler-748fbbdb59-w4k4d                 0/1     Init:0/1            0          30s   <none>      member4-worker2   <none>           <none>
karmada-static-resource-djnkn                      0/1     ContainerCreating   0          30s   <none>      member4-worker2   <none>           <none>
karmada-webhook-58d9dfdd95-qndkp                   0/1     Init:0/1            0          30s   10.12.6.6   member4-worker    <none>           <none>

3. generate the kubeconfig file, and use kubectl to access karmada-apiserver

$ kubectl get secret -n karmada-system karmada-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > ~/.kube/karmada-apiserver-temp.config
$ kubectl --kubeconfig ~/.kube/karmada-apiserver-temp.config get pp 
No resources found in default namespace.

4. register a new pull mode cluster member5

$ eval $(karmadactl token create --print-register-command  --kubeconfig ~/.kube/karmada-apiserver-temp.config --karmada-context karmada-apiserver) --kubeconfig ~/.kube/member5.config --context member5 --karmada-agent-image docker.io/karmada/karmada-agent:latest --cluster-name member5
[preflight] Running pre-flight checks
[preflight] All pre-flight checks were passed
[karmada-agent-start] Waiting to perform the TLS Bootstrap
[karmada-agent-start] Waiting to check cluster exists
[karmada-agent-start] Assign the necessary RBAC permissions to the agent
[karmada-agent-start] Waiting to construct karmada-agent kubeconfig
[karmada-agent-start] Waiting the necessary secret and RBAC
[karmada-agent-start] Waiting karmada-agent Deployment

cluster(member5) is joined successfully

$ kubectl --kubeconfig ~/.kube/karmada-apiserver-temp.config get cluster
NAME      VERSION   MODE   READY   AGE
member5   v1.31.2   Pull   True    26s

[chaosi-zju]

$ helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --set apiServer.hostNetwork=false,apiServer.hostIP=172.18.0.7,apiServer.serviceType=NodePort,apiServer.nodePort=31333

1. As for multi-node cluster, the hostIP configuration is confusing because it doesn't actually represent the host IP (just like you specifed 172.18.0.7, but your real host IP is 172.18.0.9), making it hard for users to fill in correctly and increasing the likelihood of errors.
2. The hostIP and serviceType configurations are overly coupled. The value of one field determines that of another. Incorrect installation inputs can't be easily fixed, forcing reinstallation, showing distinct limitations compared to other installation methods. (just like hostIP=127.18.0.7 and serviceType=NodePort works, but hostIP=127.18.0.7 and serviceType=ClusterIP not work)