Intermittent issues with --tag

[duglin]

In what area(s)?

What version of Knative Client?

master

What version of Knative Serving running on your cluster?

0.8.x

Issue

Maybe 10% of the time when I run this command I see this:

$ kn service update echo --revision-name=echo-v3 --tag echo-v3=test --env MSG="TEST TEST TEST"
Waiting for service 'echo' to become ready ... 
RevisionMissing: Revision "echo-v3" referenced in traffic not found.

What's odd is that it only fails some of the time, so there must be a timing issue involved here.

Will try to do more debugging but wanted to let @navidshaikh know that something's up...

[navidshaikh]

I dont think if this is something we control, seems like revision echo-v3 wasn't available when traffic spec is parsed by serving.

Also this is not re-producible easily as you mentioned.

➜ kn service create echo --image gcr.io/knative-samples/helloworld-go
Service 'echo' successfully created in namespace 'default'.
Waiting for service 'echo' to become ready ... OK

Service URL:
http://echo.default.apps-crc.testing

➜ kn service update echo --revision-name=echo-v3 --tag echo-v3=test --env MSG="TEST TEST TEST"
Waiting for service 'echo' to become ready ... OK
Service 'echo' updated in namespace 'default'.

➜ kn revision list
NAME           SERVICE   GENERATION   AGE   CONDITIONS   READY   REASON
echo-v3        echo      2            23s   4 OK / 4     True    
echo-mbszk-1   echo      1            75s   4 OK / 4     True    

➜ kn service list
NAME   URL                                    GENERATION   AGE   CONDITIONS   READY   REASON
echo   http://echo.default.apps-crc.testing   2            82s   3 OK / 3     True   

In kn, when user ran
$ kn service update echo --revision-name=echo-v3 --tag echo-v3=test --env MSG="TEST TEST TEST"
we generated spec similar to what user could've written in YAML and posted to serving. Wondering if its something to be looked at serving side.

[duglin]

I haven’t had a chance to look at the code yet, do you do two requests to the server or just one? I think if you did a request then it would always work.

[navidshaikh]

@duglin : We do only one request.

[sixolet]

Yeah I think this is a bug to send over to serving.

…

On Aug 24, 2019, at 10:39 PM, Navid Shaikh ***@***.***> wrote: @duglin : We do only one request. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[duglin]

I created this script to reproduce it w/o kn so I could open an issue on serving:

#!/bin/bash
set -ex

kubectl delete ksvc/echo || true
sleep 5

kubectl apply -f - <<EOF
apiVersion: serving.knative.dev/v1beta1
kind: Service
metadata:
  name: echo
spec:
  template:
    metadata:
      name: echo-v1
    spec:
      containers:
      - image: duglin/echo
  traffic:
  - revisionname: echo-v1
    percent: 100
EOF

sleep 5

while true ; do

val=$RANDOM

kubectl apply -f - <<EOF
apiVersion: serving.knative.dev/v1beta1
kind: Service
metadata:
  name: echo
spec:
  template:
    metadata:
      name: echo-v$val
    spec:
      containers:
      - image: duglin/echo
        env:
        - name: MSG
          value: val$RANDOM
  traffic:
  - revisionName: echo-v1
    percent: 90
  - revisionName: echo-v$val
    tag: test
    percent: 10
EOF

if kubectl get ksvc/echo -o yaml | grep "not found" ; then
  exit 1
fi

sleep 10

done

however, I noticed that after I got the error, eventually the server did end up in good state. This makes me wonder whether this is really a serving issue or a kn issue. Meaning, should kn do a better job of detecting this intermediate error state and then wait bit more for things to settle down before it decides to return an error to the user?

Of course, even if that's the case, it does make it kind of hard for any client to know when the error it's seeing is "real" vs "temporary".

@mattmoor what do you think? Should this error ever be the conditions section at all?

[sixolet]

Server contract says ready shouldn't go false if it's going to be true once it settles. Come to think of it, I seem to recall dangerd tracking an issue like this. Next I'm working at a real computer I'll go look if you don't get to it first.

…

On Aug 25, 2019, at 4:16 PM, Doug Davis ***@***.***> wrote: I created this script to reproduce it w/o kn so I could open an issue on serving: #!/bin/bash set -ex kubectl delete ksvc/echo || true sleep 5 kubectl apply -f - <<EOF apiVersion: serving.knative.dev/v1beta1 kind: Service metadata: name: echo spec: template: metadata: name: echo-v1 spec: containers: - image: duglin/echo traffic: - revisionname: echo-v1 percent: 100 EOF sleep 5 while true ; do val=$RANDOM kubectl apply -f - <<EOF apiVersion: serving.knative.dev/v1beta1 kind: Service metadata: name: echo spec: template: metadata: name: echo-v$val spec: containers: - image: duglin/echo env: - name: MSG value: val$RANDOM traffic: - revisionName: echo-v1 percent: 90 - revisionName: echo-v$val tag: test percent: 10 EOF if kubectl get ksvc/echo -o yaml | grep "not found" ; then exit 1 fi sleep 10 done however, I noticed that after I got the error, eventually the server did end up in good state. This makes me wonder whether this is really a serving issue or a kn issue. Meaning, should kn do a better job of detecting this intermediate error state and then wait bit more for things to settle down before it decides to return an error to the user? Of course, even if that's the case, it does make it kind of hard for any client to know when the error it's seeing is "real" vs "temporary". @mattmoor what do you think? Should this error ever be the conditions section at all? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[duglin]

Just FYI so we have it, when the error happens the status section looks like this:

status:
  address:
    url: http://echo.default.svc.cluster.local
  conditions:
  - lastTransitionTime: "2019-08-25T23:45:25Z"
    message: The Configuration is still working to reflect the latest desired specification.
    reason: OutOfDate
    status: Unknown
    type: ConfigurationsReady
  - lastTransitionTime: "2019-08-25T23:45:25Z"
    message: Revision "echo-v392" referenced in traffic not found.
    reason: RevisionMissing
    status: "False"
    type: Ready
  - lastTransitionTime: "2019-08-25T23:45:25Z"
    message: Revision "echo-v392" referenced in traffic not found.
    reason: RevisionMissing
    status: "False"
    type: RoutesReady
  latestCreatedRevisionName: echo-v16289
  latestReadyRevisionName: echo-v16289

[duglin]

@sixolet can you point me to where it says that in the RTC? I can't seem to find it.

@mattmoor @dgerd is this a server-side issue or should the cli be checking something else to know when things are settled down?

[mattmoor]

knative/serving#4173 ?

[duglin]

Sure sounds like it.

[duglin]

knative/serving#5547 is in now - will retest soon

[dgerd]

@duglin Let me know if you have any further problems here.

[navidshaikh]

/close

@duglin tested using the script you provided above, 10+ updates to service and it worked fine. Closing the issue, please re-open if you observe otherwise.
Thanks @dgerd for the fix!

[knative-prow-robot]

@navidshaikh: Closing this issue.

In response to this:

/close

@duglin tested using the script you provided above, 10+ updates to service and it worked fine. Closing the issue, please re-open if you observe otherwise.
Thanks @dgerd for the fix!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.