MNIST Example Doesn't Work #578

Tabrizian · 2019-06-25T13:44:03Z

I tried the local version of MNIST example and kustomize fails with the error below:

Error: var '{batchSize ~G_~V_ConfigMap {data.batchSize}}' cannot be mapped to a field in the set of known resources

The text was updated successfully, but these errors were encountered:

issue-label-bot · 2019-06-25T13:44:05Z

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.88. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

jinchihe · 2019-06-26T14:06:46Z

Could you please provide more information, such as kustomize version? k8s version, and which step the problem happened? @Tabrizian

jinchihe · 2019-06-26T14:34:47Z

You can try to check the batchSize is set by running, or list the training/local/kustomization.yaml file here.

# grep -i batchSize kustomization.yaml
  - batchSize=100

davidtwomey · 2019-06-26T14:58:09Z

@Tabrizian @jinchihe
Was having the same issue using kustomization latest release (v3.0.0 pre) as per tutorial:
https://codelabs.developers.google.com/codelabs/kubeflow-introduction/#0

Got kustomization build . to work (for GCS) by

Switching to explicit kustomization v2.0.0
Changing from kustomize edit set image training-image=$TRAIN_PATH to
kustomize edit set image training-image=$TRAIN_PATH:latest
i.e. appending image tag

I tried the local version of MNIST example and kustomize fails with the error below:
Error: var '{batchSize ~G_~V_ConfigMap {data.batchSize}}' cannot be mapped to a field in the set of known resources

jinchihe · 2019-06-26T15:15:48Z

@davidtwomey Yes, I tested kustomization 2.0.3, that should work fine. And for you mentioned "$TRAIN_PATH:latest" , In the readme file, the $TRAIN_PATH should include tag as below description:

DOCKER_URL=docker.io/reponame/mytfmodel:tag

davidtwomey · 2019-06-26T15:27:33Z

@jinchihe
Yep - the $TRAIN_PATH was just a heads up for those following via the Google colabs tutorial
where train path did not have a tag
GCP TRAIN_PATH=us.gcr.io/$PROJECT_ID/kubeflow-train

Tabrizian · 2019-06-27T02:17:20Z

kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}                            Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

kustomize version
Version: {KustomizeVersion:3.0.0-pre GitCommit:ae5ebccec77a8691017e11b9b67cd167f0e27856 BuildDate:2019-06-24T01:14:40Z GoOs:linux GoArch:amd64}

The problem happened when I tried to kustomize build ..

Tabrizian · 2019-06-27T02:24:00Z

@jinchihe

grep -i batchSize kustomization.yaml
  - batchSize=100

outputs the batchSize=100 value.

jinchihe · 2019-06-27T02:26:07Z

@jinchihe
grep -i batchSize kustomization.yaml
  - batchSize=100
outputs the batchSize=100 value.

So I guess that should be caused your too old kustomize, what's kustomize version?

Tabrizian · 2019-06-27T02:36:28Z

v3.0.0-pre1

Tabrizian · 2019-06-27T02:39:49Z

I tried version 2.1.0 and it works. Probably it is kustomize problem.

Tabrizian · 2019-06-27T02:41:35Z

Sorry, it didn't work. In version 2.1.0 generates the same error.

Error: var '{batchSize ~G_~V_ConfigMap {data.batchSize}}' cannot be mapped to a field in the set of known resources

parallelo · 2019-06-27T03:22:58Z

I was hitting the same errors earlier today.

I don't have a root cause yet, but here are some observations:

seems to work with kustomize v2.0.3 and before
seems to fail with kustomize v2.1.0 and later

jinchihe · 2019-06-27T03:33:14Z

@parallelo Oh.. Thanks! so seems that's kustomize issue? Need to check what's changed by new kustomze.
@Tabrizian You can try with kustomize v2.0.3, I'm sure that work fine with v2.0.3

Tabrizian · 2019-06-27T03:48:22Z

Very very strange! It works with v2.0.3.

jinchihe · 2019-07-02T02:44:09Z

I think that may be a bug in new kustomize, logged one issue the kustomize to trace: kubernetes-sigs/kustomize#1295

aginpatrick · 2019-07-04T14:02:32Z

I'm having the same issue with kustomize version 2.1.0. Can you tell me how to downgrade to 2.0.3? (kubectl version is 1.8.6). Thanks

Tabrizian · 2019-07-04T14:15:24Z

Download it from here and copy it to your $PATH.

aginpatrick · 2019-07-04T14:22:14Z

Thanks for your help @Tabrizian! Sorry I'm not very familiar with this process. I don't see any download button on the page. Do you mean download the binary that I see at the bottom of the page? Isn't it possible to do something like brew install kustomize@2.0.3? (I'm on MacOS)

Tabrizian · 2019-07-04T15:59:21Z

Download from this link https://github.com/kubernetes-sigs/kustomize/releases/download/v2.0.3/kustomize_2.0.3_darwin_amd64.
mv kustomize_2.0.3_darwin_amd64 /usr/bin/kustomize

aginpatrick · 2019-07-04T16:40:59Z

Thank you @Tabrizian. Unfortunately I get now the following error with version 2.0.3:

`Error: rawResources failed to read Resources: Load from path ../base failed: '../base' must be a file (got d='/Users/me/repos/examples/mnist/training/base')

aginpatrick · 2019-07-04T17:01:52Z

Reading this: kubernetes-sigs/kustomize#766
Changed resources for bases in kustomization.yaml
build is working now
(but I get another error later when I try to get the web-ui service with `kubectl get service web-ui: service is not found, I don't know yet if it's related or not to the kustomize downgrading)

Tabrizian · 2019-07-05T02:12:36Z

I had similar problem. It was fixed with a fresh clone of the repository. Try it and let me know if it was not fixed.

jinchihe · 2019-07-05T02:24:17Z

@Tabrizian I just send a PR #584 to recommand user to usr kustomize v2.0.3 to avoid the problem. marked fixes the problem, but the ticket is not closed automatically, that's strange... Suggest close the ticket now if you agree. Thanks.

Tabrizian · 2019-07-05T02:45:12Z

Thanks! Sure, I will close the issue.

aginpatrick · 2019-07-05T14:31:33Z

@Tabrizian, what repo exactly? If it's https://github.com/kubeflow/examples.git, I cloned it two days ago so I would be surprised if it is the solution to my problem. Anyway, I will try it again and let you know. Thank you again.

vahidtz · 2019-09-23T18:14:14Z

@jinchihe
Yep - the $TRAIN_PATH was just a heads up for those following via the Google colabs tutorial
where train path did not have a tag
GCP TRAIN_PATH=us.gcr.io/$PROJECT_ID/kubeflow-train

I used kustomize version 2.0.3 and included the tag for the training image but still have the base folder issue, do you any other hint?
kustomize edit set image training-image=$TRAIN_PATH:latest
kustomize build .

Error: rawResources failed to read Resources: Load from path ../base failed: '../base' must be a file (got d='/h
/examples/mnist/training/base')

kustomize version
Version: {KustomizeVersion:2.0.3 GitCommit:a6f65144121d1955266b0cd836ce954c04122dc8 BuildDate:2019-03-05T20:37:4
2Z GoOs:linux GoArch:amd64}

ptone · 2020-01-11T02:02:51Z

still getting this with v3.5.3
{Version:kustomize/v3.5.3 GitCommit:5ba90fe5ef1dc4599e359edd41d1d0e6373b247d BuildDate:2019-12-17T21:57:37Z GoOs:linux GoArch:amd64}

Geoyi · 2020-02-01T19:06:59Z

still getting the same error mentioned above with version 3.5.4

{Version:3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-17T14:23:25+00:00 GoOs:darwin GoArch:amd64}

issue-label-bot bot added the kind/bug label Jun 25, 2019

Tabrizian closed this as completed Jun 27, 2019

Tabrizian reopened this Jun 27, 2019

This was referenced Jul 2, 2019

[codelab-intro-mnist-feedback]: #583

Closed

recomment using kustomize v2.0.3 for mnist #584

Merged

Tabrizian closed this as completed Jul 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MNIST Example Doesn't Work #578

MNIST Example Doesn't Work #578

Tabrizian commented Jun 25, 2019

issue-label-bot bot commented Jun 25, 2019

jinchihe commented Jun 26, 2019

jinchihe commented Jun 26, 2019

davidtwomey commented Jun 26, 2019 •

edited

Loading

jinchihe commented Jun 26, 2019

davidtwomey commented Jun 26, 2019

Tabrizian commented Jun 27, 2019 •

edited

Loading

Tabrizian commented Jun 27, 2019 •

edited

Loading

jinchihe commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

parallelo commented Jun 27, 2019

jinchihe commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

jinchihe commented Jul 2, 2019

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 4, 2019 •

edited

Loading

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 4, 2019

aginpatrick commented Jul 4, 2019

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 5, 2019

jinchihe commented Jul 5, 2019

Tabrizian commented Jul 5, 2019

aginpatrick commented Jul 5, 2019

vahidtz commented Sep 23, 2019 •

edited

Loading

ptone commented Jan 11, 2020

Geoyi commented Feb 1, 2020

MNIST Example Doesn't Work #578

MNIST Example Doesn't Work #578

Comments

Tabrizian commented Jun 25, 2019

issue-label-bot bot commented Jun 25, 2019

jinchihe commented Jun 26, 2019

jinchihe commented Jun 26, 2019

davidtwomey commented Jun 26, 2019 • edited Loading

jinchihe commented Jun 26, 2019

davidtwomey commented Jun 26, 2019

Tabrizian commented Jun 27, 2019 • edited Loading

Tabrizian commented Jun 27, 2019 • edited Loading

jinchihe commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

parallelo commented Jun 27, 2019

jinchihe commented Jun 27, 2019

Tabrizian commented Jun 27, 2019

jinchihe commented Jul 2, 2019

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 4, 2019 • edited Loading

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 4, 2019

aginpatrick commented Jul 4, 2019

aginpatrick commented Jul 4, 2019

Tabrizian commented Jul 5, 2019

jinchihe commented Jul 5, 2019

Tabrizian commented Jul 5, 2019

aginpatrick commented Jul 5, 2019

vahidtz commented Sep 23, 2019 • edited Loading

ptone commented Jan 11, 2020

Geoyi commented Feb 1, 2020

davidtwomey commented Jun 26, 2019 •

edited

Loading

Tabrizian commented Jun 27, 2019 •

edited

Loading

Tabrizian commented Jun 27, 2019 •

edited

Loading

Tabrizian commented Jul 4, 2019 •

edited

Loading

vahidtz commented Sep 23, 2019 •

edited

Loading