recomment using kustomize v2.0.3 for mnist

[jinchihe]

Recently someones hit one problem while running mnist example with new kustomize (v2.1.0), such as #583 and #578, after checked the problem, I think that's should be a issue in new kustomize, logged ticket to trace in kustomize.

But avoid user stepping on the pit, the PR is to add note to recomment user to use kustomize 2.0.3. May enhance later if the kustomzie issue fixed.

fixes: #583 #578

---

This change is 

[jlewi]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jinchihe]

/retest

[jinchihe]

Tested failed due to

error: unable to recognize "STDIN": no matches for kind "TFJob" in version "kubeflow.org/v1beta2

Seems the TFJob is upgraded in the test cluster? will confirm later.

[jinchihe]

That's stange, checked the code, the version v1beta2 of TFJob should exist in the cluster, check some code to show the TFJob version for debug... Thanks.

[jinchihe]

From the debug information, the tfjobs is not installed in the testing cluster. As below error:

Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io “tfjobs.kubeflow.org” not found

@jlewi Do you know what's changed in the testing cluster? Anyone can help to check/fix this? Thanks.

[jlewi]

What cluster are you using?
Is it kf-vmaster-XX
Those clusters are now deploying from master which might be using TFJob v1.

[jlewi]

I checked v-kfmaster-n01
and it looks like TFJob is defined.

See attached
tfjob.yaml.txt

Its v1.

[jlewi]

Opened #588 to give you access to the clusters.

[jinchihe]

@jlewi Many thanks. Yes, that works fine with the cluster!

[hougangliu]

/lgtm

[aginpatrick]

@jinchihe @jlewi would it be useful to add that kustomize v2 blocks relative paths (it's no more true with v3). So kustomization.yaml cannot contain "resources: -../bases" for example. I think that substitute resources for bases is the solution in this case.

[aginpatrick]

@jinchihe @jlewi btw, does using kustomize v2.0.3 has implications for kubectl version? I mean, is v2.0.3 compatible with latest kubectl or one has to use older version?

[jinchihe]

@aginpatrick kustomize should have no implications with kubectl version. But kustomize has been intergrated in the kubectl, see https://kubernetes.io/blog/2019/03/25/kubernetes-1-14-release-announcement/. kustomize v2.0.3 is available in kubectl v1.15.