[codelab-intro-mnist-feedback]:build #609

wanglj-lanzer · 2019-07-25T06:24:09Z

kustomize build .

The text was updated successfully, but these errors were encountered:

issue-label-bot · 2019-07-25T06:24:11Z

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.83. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

miguelvcsoares · 2019-07-25T16:11:19Z

I encountered the same issue. What can we do to solve it?

jinchihe · 2019-07-26T05:32:15Z

@wanglj-lanzer Which version of kustomize you used? we hit some problems due to kustomize bug, and seuggested (noted) using kustomize v2.0.3 in README file.

And which platform you tried, GCP or on-prem cluster? Thanks.

wanglj-lanzer · 2019-07-26T05:43:22Z

GCP

I installed the kustomize via go get sigs.k8s.io/kustomize/cmd/kustomize

jinchihe · 2019-07-26T05:51:41Z

See notes in the readme file
Note: kustomize v2.0.3 is recommented since the problem in kustomize v2.1.0

Could you please try this by using v2.0.3 and following the Readme file? New kustomize has bug here.

wanglj-lanzer · 2019-07-26T18:40:27Z

new errors after reinstall kustomize as the recommended version

Error: rawResources failed to read Resources: Load from path ../base failed: '../base' must be a file (got d='~/examples/mnist/training/base')

andrewcantos · 2019-07-29T07:49:15Z

I have also experienced same issues as @wanglj-lanzer – both on latest kustomize and v2.0.3.

jinchihe · 2019-07-30T01:29:12Z

Following the README in the repo, that should work fine. Thanks.

andrewcantos · 2019-07-30T07:21:10Z

Unfortunately it doesn't work as expected on my machine. I have ensured I am running v2.0.3 using cloned latest from (this) examples repo. I have followed README instructions and error:

Error: rawResources failed to read Resources: Load from path ../base failed: '../base' must be a file (got d='~/kubeflow_examples/mnist/training/base')

Note: I ran this from mnist/training/GCS directory.

I was able to successfully build from local directory.

jinchihe · 2019-07-30T07:45:49Z

@andrewcantos @wanglj-lanzer I'm not sure which steps has the problem? But that works fine on my env, below is a example, thanks.:

[root@jinchi1 GCS]# kustomize version
Version: {KustomizeVersion:2.0.3 GitCommit:a6f65144121d1955266b0cd836ce954c04122dc8 BuildDate:2019-03-05T20:37:42Z GoOs:linux GoArch:amd64}
[root@jinchi1 GCS]# BUCKET=distributed-test
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=name=mnist-train-dist
[root@jinchi1 GCS]# kustomize edit set image training-image=docker.io/ibmlei/mykubeflow:0.1
[root@jinchi1 GCS]# ../base/definition.sh --numPs 1 --numWorkers 2
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=trainSteps=200
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=batchSize=100
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=learningRate=0.01
[root@jinchi1 GCS]# MODEL_PATH=my-model
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=modelDir=gs://${BUCKET}/${MODEL_PATH}
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=exportDir=gs://${BUCKET}/${MODEL_PATH}/export
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=secretName=user-gcp-sa
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=secretMountPath=/var/secrets
[root@jinchi1 GCS]# kustomize edit add configmap mnist-map-training --from-literal=GOOGLE_APPLICATION_CREDENTIALS=/var/secrets/user-gcp-sa.json
[root@jinchi1 GCS]# kustomize build . |kubectl apply -f -
configmap/mnist-map-training-gfththd74g created
tfjob.kubeflow.org/mnist-train-dist created

ricoms · 2019-08-06T02:09:17Z

I'm following this tutorial. I'm pretty sure I followed the instructions correctly and I reached the same problem at step kustomize build .

$ kustomize build .
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1beta2_TFJob|$(trainingName)

I installed kustomize from "Quickly curl the latest" here

my version is as below:
Version: {KustomizeVersion:3.1.0 GitCommit:95f3303493fdea243ae83b767978092396169baf BuildDate:2019-07-26T18:11:16Z GoOs:linux GoArch:amd64}

jinchihe · 2019-08-06T02:19:00Z

@ricoms use kustomize v2.0.3 please, new kustomize has bugs here.

ricoms · 2019-08-06T18:22:09Z

using v2.0.3 I fell on the issue you opened @jinchihe #1029 when I applied kustomize edit set image training-image=$IMAGE_PATH

I created a tag with export TAG=$(date +"%m-%d-%Y") and add it to the IMAGE_PATH like this export IMAGE_PATH=us.gcr.io/$PROJECT_ID/kubeflow-train:$TAG.

Then I was able to apply kustomize build . | kubectl apply -f - with success, the respose was a yaml file and below:

configmap/mnist-map-training-4c92g6hh6b unchanged
tfjob.kubeflow.org/my-train-1 unchanged

But the original "bug" seems so "simple"... the problem appears to be the trainingName variable, as pointed here. The kustomization.yaml requires trainingName variable and it is not being set anywhere, it seems. I would like to solve this problem by upgrading the tutorial to kustomize 3.1.0. Is it very hard? Could someone point me to the problem and where could I help?

jinchihe · 2019-08-07T00:10:19Z

@ricoms See here kubernetes-sigs/kustomize#1295 I think that has been fixed, and code may be merged in 3.x.

lucastheis · 2019-08-14T17:08:37Z

Calling kustomize_3.x.x edit [...] once replaces

bases:
- ../base

with

resources:
- ../base

which leads to Error: rawResources failed to read Resources: Load from path ../base failed. Reverting that change and using kustomize_2.0.3 fixed it for me.

sarika-pst · 2019-09-20T13:42:43Z

Getting the same error
kustomize build . |kubectl apply -f - Error: no matches for OriginalId extensions_v1beta1_Deployment|~X|$(svcName); no matches for CurrentId extensions_v1beta1_Deployment|~X|$(svcName); failed to find unique target for patch extensions_v1beta1_Deployment|$(svcName) error: no objects passed to apply
Any solution for this?
Thanks in advance.

jinchihe · 2019-09-22T09:56:58Z

@aicdp1010 try it using kustomize v2.0.3 please.

sarika-pst · 2019-09-23T12:47:05Z

Above error is resolved using v2.0.3 but now I am getting

$ kustomize build . |kubectl apply -f -
configmap/mnist-map-training-2fg9c7886m created
error: unable to recognize "STDIN": no matches for kind "TFJob" in version "kubeflow.org/v1beta2"

I am new to this, Is the issue related to kubernetes cluster. Please guide me.
Thank you.

jinchihe · 2019-09-23T13:51:19Z

@aicdp1010 please check your cluster, what the TFJob version installed in your cluster?

shreya-verma19 · 2019-10-07T10:56:47Z

using v2.0.3 I fell on the issue you opened @jinchihe #1029 when I applied kustomize edit set image training-image=$IMAGE_PATH

I created a tag with export TAG=$(date +"%m-%d-%Y") and add it to the IMAGE_PATH like this export IMAGE_PATH=us.gcr.io/$PROJECT_ID/kubeflow-train:$TAG.

Then I was able to apply kustomize build . | kubectl apply -f - with success, the respose was a yaml file and below:
configmap/mnist-map-training-4c92g6hh6b unchanged
tfjob.kubeflow.org/my-train-1 unchanged
But the original "bug" seems so "simple"... the problem appears to be the trainingName variable, as pointed here. The kustomization.yaml requires trainingName variable and it is not being set anywhere, it seems. I would like to solve this problem by upgrading the tutorial to kustomize 3.1.0. Is it very hard? Could someone point me to the problem and where could I help?

Hi I tried to add trainingName but it already existed
kustomize edit add configmap mnist-map-training --from-literal=trainingName=test_kubeflow
Error: cannot add key trainingName, another key by that name already exists: map[GOOGLE_APPLICATION_CREDENTIALS:/var/secrets/user-gcp-sa.json batchSize:100 exportDir:gs://my_bucket_kubeflow/my-model/export learningRate:0.01 modelDir:gs://my_bucket_kubeflow/my-model name:my-train-1 secretMountPath:/var/secrets secretName:user-gcp-sa trainSteps:200 trainingName:test_kubeflow]

But when I am again trying to build it gives error

cloudshell:~/examples/mnist/training/GCS$ kustomize build
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1beta2_TFJob|$(trainingName)

jinchihe · 2019-10-09T02:02:33Z

@shreya-verma19 you executed twice? if yes, need to remove the frist one, seems kustomize does not support rewriting the value.

bruscar010 · 2019-10-31T19:17:32Z

hi @jinchihe

Following this guide here

configmap/mnist-map-training-45h47275m7 unchanged error: unable to recognize "STDIN": no matches for kind "TFJob" in version "kubeflow.org/v1beta2"
I've been reading through a couple of these threads and I can't find a solution. Looking for any advice on what I can do

Version: {KustomizeVersion:2.0.3 GitCommit:a6f65144121d1955266b0cd836ce954c04122dc8 BuildDate:2019-03-05T20:37:42Z GoOs:linux GoArch:amd64}

I ran kubectl describe crd tfjobs.kubeflow.org and this is what returned

Name:         tfjobs.kubeflow.org
Namespace:    
Labels:       app.kubernetes.io/component=tfjob
              app.kubernetes.io/instance=tf-job-crds-v0.7.0
              app.kubernetes.io/managed-by=kfctl
              app.kubernetes.io/name=tf-job-crds
              app.kubernetes.io/part-of=kubeflow
              app.kubernetes.io/version=v0.7.0
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"labels":{"app.kubernetes.io/c...
API Version:  apiextensions.k8s.io/v1beta1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2019-10-29T23:53:03Z
  Generation:          1
  Resource Version:    2620
  Self Link:           /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/tfjobs.kubeflow.org
  UID:                 3eb96b55-faa7-11e9-9ab9-42010a840fdc
Spec:
  Additional Printer Columns:
    JSON Path:  .status.conditions[-1:].type
    Name:       State
    Type:       string
    JSON Path:  .metadata.creationTimestamp
    Name:       Age
    Type:       date
  Conversion:
    Strategy:  None
  Group:       kubeflow.org
  Names:
    Kind:       TFJob
    List Kind:  TFJobList
    Plural:     tfjobs
    Singular:   tfjob
  Scope:        Namespaced
  Subresources:
    Status:
  Validation:
    openAPIV3Schema:
      Properties:
        Spec:
          Properties:
            Tf Replica Specs:
              Properties:
                Chief:
                  Properties:
                    Replicas:
                      Maximum:  1
                      Minimum:  1
                      Type:     integer
                PS:
                  Properties:
                    Replicas:
                      Minimum:  1
                      Type:     integer
                Worker:
                  Properties:
                    Replicas:
                      Minimum:  1
                      Type:     integer
  Version:                      v1
  Versions:
    Name:     v1
    Served:   true
    Storage:  true
Status:
  Accepted Names:
    Kind:       TFJob
    List Kind:  TFJobList
    Plural:     tfjobs
    Singular:   tfjob
  Conditions:
    Last Transition Time:  2019-10-29T23:53:03Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  <nil>
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>

jinchihe · 2019-11-01T00:51:13Z

You tfjob CRD is V1, so we need to update the tfjobs version to v1.

bruscar010 · 2019-11-01T08:38:11Z

@jinchihe is that as simple as changing one of the yaml files from v1beta2 to v1?

jinchihe · 2019-11-01T23:52:37Z

@FrancisLennon17 I think need to update all yaml files, I will update this once get chance. thanks

tarrade · 2019-11-19T13:23:40Z

I got the same issues and I did the following (as mention aboved):

use PATH=${WORKING_DIR}/bin:$PATH so it will start with KustomizeVersion:2.0.3 and not some other version you have installed
in mnist/training/GCS/kustomization.yam replace: "ressources" by "bases" and it should look like:

bases: 
- ../base

you can also look at the correct YAML for serving as an example: mnist/serving/GCS/kustomization.yam

Then everything is working fine on my side.

plaffitte · 2019-11-28T17:39:38Z

@jinchihe I tried replacing all v1beta2 with v1 in all .yaml files but I get the same issue with v1:

unable to recognize "STDIN": no matches for kind "TFJob" in version "kubeflow.org/v1"

dhfromkorea · 2019-12-06T07:18:49Z

I had the same issue as reported by @plaffitte.

paulkarayan · 2019-12-24T00:19:23Z

based on @lucastheis 's situation, i downgraded to older Kustomize package
(you can do this will brew, following https://www.fernandomc.com/posts/brew-install-legacy-hugo-site-generator/)
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/b4e8be2ffb5b9552901d802911dbae522654cf1e/Formula/kustomize.rb

then if i edit GCS/kustomization.yaml to replace "resources" with "bases" e.g.

images:
- name: training-image
  newTag: latest
bases:
- ../base
configMapGenerator:

etc

then this works. really obnoxious.

jtfogarty · 2020-01-16T16:52:49Z

/area example/mnist
/area kustomize
/priority p2

stale · 2020-04-15T20:56:56Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

issue-label-bot bot added the kind/bug label Jul 25, 2019

wanglj-lanzer mentioned this issue Jul 26, 2019

[codelab-intro-mnist-feedback]: #607

Closed

ricoms mentioned this issue Aug 6, 2019

Test and update the GCP end to end Kubeflow tutorial for Kubeflow v0.6 kubeflow/website#844

Closed

jinchihe mentioned this issue Aug 20, 2019

[codelab-intro-mnist-feedback]: #622

Closed

fenglixa mentioned this issue Nov 25, 2019

unable to build kustomize for mnist example #681

Closed

k8s-ci-robot added area/example/mnist area/kustomize priority/p2 labels Jan 16, 2020

stale bot added the lifecycle/stale label Apr 15, 2020

stale bot closed this as completed Apr 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[codelab-intro-mnist-feedback]:build #609

[codelab-intro-mnist-feedback]:build #609

wanglj-lanzer commented Jul 25, 2019

issue-label-bot bot commented Jul 25, 2019

miguelvcsoares commented Jul 25, 2019

jinchihe commented Jul 26, 2019

wanglj-lanzer commented Jul 26, 2019

jinchihe commented Jul 26, 2019 •

edited

Loading

wanglj-lanzer commented Jul 26, 2019

andrewcantos commented Jul 29, 2019

jinchihe commented Jul 30, 2019

andrewcantos commented Jul 30, 2019

jinchihe commented Jul 30, 2019 •

edited

Loading

ricoms commented Aug 6, 2019 •

edited

Loading

jinchihe commented Aug 6, 2019

ricoms commented Aug 6, 2019 •

edited

Loading

jinchihe commented Aug 7, 2019

lucastheis commented Aug 14, 2019

sarika-pst commented Sep 20, 2019

jinchihe commented Sep 22, 2019

sarika-pst commented Sep 23, 2019

jinchihe commented Sep 23, 2019

shreya-verma19 commented Oct 7, 2019

jinchihe commented Oct 9, 2019

bruscar010 commented Oct 31, 2019 •

edited

Loading

jinchihe commented Nov 1, 2019

bruscar010 commented Nov 1, 2019

jinchihe commented Nov 1, 2019

tarrade commented Nov 19, 2019

plaffitte commented Nov 28, 2019

dhfromkorea commented Dec 6, 2019

paulkarayan commented Dec 24, 2019 •

edited

Loading

jtfogarty commented Jan 16, 2020

stale bot commented Apr 15, 2020

[codelab-intro-mnist-feedback]:build #609

[codelab-intro-mnist-feedback]:build #609

Comments

wanglj-lanzer commented Jul 25, 2019

issue-label-bot bot commented Jul 25, 2019

miguelvcsoares commented Jul 25, 2019

jinchihe commented Jul 26, 2019

wanglj-lanzer commented Jul 26, 2019

jinchihe commented Jul 26, 2019 • edited Loading

wanglj-lanzer commented Jul 26, 2019

andrewcantos commented Jul 29, 2019

jinchihe commented Jul 30, 2019

andrewcantos commented Jul 30, 2019

jinchihe commented Jul 30, 2019 • edited Loading

ricoms commented Aug 6, 2019 • edited Loading

jinchihe commented Aug 6, 2019

ricoms commented Aug 6, 2019 • edited Loading

jinchihe commented Aug 7, 2019

lucastheis commented Aug 14, 2019

sarika-pst commented Sep 20, 2019

jinchihe commented Sep 22, 2019

sarika-pst commented Sep 23, 2019

jinchihe commented Sep 23, 2019

shreya-verma19 commented Oct 7, 2019

jinchihe commented Oct 9, 2019

bruscar010 commented Oct 31, 2019 • edited Loading

jinchihe commented Nov 1, 2019

bruscar010 commented Nov 1, 2019

jinchihe commented Nov 1, 2019

tarrade commented Nov 19, 2019

plaffitte commented Nov 28, 2019

dhfromkorea commented Dec 6, 2019

paulkarayan commented Dec 24, 2019 • edited Loading

jtfogarty commented Jan 16, 2020

stale bot commented Apr 15, 2020

jinchihe commented Jul 26, 2019 •

edited

Loading

jinchihe commented Jul 30, 2019 •

edited

Loading

ricoms commented Aug 6, 2019 •

edited

Loading

ricoms commented Aug 6, 2019 •

edited

Loading

bruscar010 commented Oct 31, 2019 •

edited

Loading

paulkarayan commented Dec 24, 2019 •

edited

Loading