v1beta1 failed mutation requests

[dgerd]

In what area(s)?

/area API

What version of Knative?

0.7
HEAD

Expected Behavior

Updating Knative Services through the v1beta1 endpoint works.

Actual Behavior

Editing a Knative Service objects returns an error from the API Server.

$ kubectl edit ksvc runtime-test-image
error: services.serving.knative.dev "runtime-test-image" could not be patched: Internal error occurred: no kind "Service" is registered for version "serving.knative.dev/v1beta1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:51"

Steps to Reproduce the Problem

Versions:

Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.8-dispatcher", GitCommit:"1215389331387f57594b42c5dd024a2fe27334f8", GitTreeState:"clean", BuildDate:"2019-05-13T18:09:56Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.24", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-06-22T02:44:12Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

This is how my cluster got into this state. Not sure which parts are relevant.

1. Create Knative Cluster with just the v1alpha1 endpoint (such as Knative 0.6)
2. Create Knative Services in the cluster
3. Update cluster to either HEAD or Knative 0.7 which enables the v1beta1 endpoint
4. Attempt to edit existing services

Workaround

1. Use Kubernetes 1.14+ (This has not been verified)
2. On your first edit of your Knative Service also update the apiVersion to v1alpha1. After performing this once future edits should work correctly.

apiVersion: serving.knative.dev/v1beta1 <--- Change this to v1alpha1
...
spec:
  template:
    spec:
      containers:
      - image: gcr.io/dangerd-dev/runtime:v2.0 <--- While updating your spec

Related

kubernetes/kubernetes#73752
kubernetes/kubernetes#74154

[dgerd]

Easiest way I have reproduced this.

1. Create a new Service
2. Run the command below (Removes v1beta1 endpoint and then adds it back)

sed -e '28,30d' config/300-service.yaml | kubectl apply -f - && kubectl apply -f config/300-service.yaml

3. Attempt to edit your Service

[dgerd]

Validated that I cannot reproduce the bug in a 1.14 cluster

[mattmoor]

@dgerd We should talk about this at the API WG tomorrow. Hopefully we'll know some more by then, but we should try to make some sort of determination on this by next Tuesday in case we want to patch 0.7

[mattmoor]

Here's how this is shaping up in my head....

By dark launching v1beta1 in 0.7 we bumped our minimum required Kubernetes version from 1.11 to 1.14 (the version where the fix landed), this was unintentional.

The "fix" for this is a relatively simple config-only change: strip these lines, and disable some tests. In a perfect world we'd have the operator already, and we could optionally enable v1beta1 in 1.14+ clusters, but alas that's not where we are today.

So I think we have a few options:

1. Rollback the endpoint changes and disable tests,
2. Release two version of the YAML (pre-/post-1.14)
3. Move our minimum version to 1.14

At this point, I think 1.14 is far too new for 3. I don't like 1. because I think the messaging should be that things are beta and should be treated as such, even if Kubernetes struggles to support it 🙄.

So I think 2. is probably my favorite of bad options, even though it adds some debt for us to carry until we have an operator or a minimum version of 1.14.

🤷‍♂

[dgerd]

WG Notes:

• Versioned clients -- what happens?
• Hard cut over upgrade -- Is it even possible? Does it work?

[steren]

Can you detail what "2. Release two version of the YAML (pre-/post-1.14)" means exactly?

[dgerd]

2. Release two version of the YAML (pre-/post-1.14)

The idea would be to release something like:

• release-0.7-pre1.14.yaml
• release-0.7-1.14.yaml

instead of our normal:

release-0.7.yaml

The diff between the two release YAMLs would be the addition of the "v1beta1" endpoint in the 1.14 version. The 1.14 version would have these three lines (L28-30) in each of the CRD objects to enable the endpoint: https://github.com/knative/serving/blob/master/config/300-service.yaml#L28

The idea is that:

1. We can continue to use the v1beta1 endpoint in our e2e and conformance tests to gain confidence as we develop Knative 0.8 by setting up a 1.14 test environment.
2. Users and vendors can opt to use the v1beta1 endpoint on newer clusters.
3. K8s 1.11-1.13 clusters will receive all the new features minus the new endpoint

The downsides as discussed in the API working group meeting is:

1. Complicates documentation as your experience depends now on Knative + K8s version
2. Behavior might be unexpected when using samples or different clients

We did not reach a conclusion between option 1 and option 2 in the meeting. There is also likely some space between these options (i.e. setup 1.14 test cluster with v1beta1, but don't create a release yaml for it).

[dgerd]

Versioned clients -- what happens?

I tested using kn to update a broken service. Using kn works to update the Service and 'fixes' it so future kubectl edits work. This seems to confirm our hypothesis that using versioned clients succeeds and is very likely why we did not catch this earlier in our upgrade tests.

Example:

$ kubectl edit ksvc
error: services.serving.knative.dev "runtime-test-image" could not be patched: Internal error occurred: no kind "Service" is registered for version "serving.knative.dev/v1beta1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:51"

$ ./kn service update runtime-test-image --limits-cpu 1000m
Service 'runtime-test-image' updated in namespace 'default'.

$ kubectl edit ksvc
service.serving.knative.dev/runtime-test-image edited

[dprotaso]

@dgerd Out of curiosity does a versioned v1beta1 client work? (Assuming no)

[dgerd]

I have a feeling it is not going to work but I will try to add it in our upgrade tests and give it a go.

…

On Wed, Jun 26, 2019 at 1:06 PM Dave Protasowski ***@***.***> wrote: @dgerd <https://github.com/dgerd> Out of curiosity does a versioned v1beta1 client work? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4533?email_source=notifications&email_token=AKLJ6MQ2MZKXEH35EHL4DC3P4PD27A5CNFSM4H3NCB4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYUVIEY#issuecomment-506024979>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKLJ6MVMQX63N2QQSDUJGQ3P4PD27ANCNFSM4H3NCB4A> .

[dgerd]

The current thought on this is:

1. Release both a 0.7.1 "beta" yaml and an "alpha" yaml. This lets users and vendors choose which API versions to enable. The "beta" yaml will have both v1alpha1 and v1beta1 endpoints enabled. The "alpha" yaml should work on all K8s versions 1.11+ and "beta" yaml should work on all K8s versions 1.14+.
2. The "default" CRDs (i.e. ko apply -f config/) will be the v1alpha1 crds
3. We will revert all knative documentation and samples to use v1alpha1 as the "apiVersion". This provides the most interoperability and portability as it will work against all Knative 0.7.1 installs.
4. We will add additional information about the "alpha" and "beta" endpoints and yamls in the Knative Serving documentation.

As we approach the release of Knative 0.8 we should re-evaluate the need to ship separate YAMLs based upon the adoption of Kubernetes 1.14 across users and cloud vendors.

[dgerd]

0.7.1 has been cut. Closing this issue.