Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate Helm values.yaml #10664

Merged
merged 1 commit into from
Sep 21, 2020

Conversation

flvndh
Copy link
Contributor

@flvndh flvndh commented Aug 31, 2020

Tackle #10634

@boring-cyborg boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Aug 31, 2020
@mik-laj
Copy link
Member

mik-laj commented Aug 31, 2020

We have some errors Can you look at it?

2020-08-31T16:29:32.9918043Z helm install airflow . --namespace airflow --set defaultAirflowRepository=apache/airflow --set images.airflow.repository=apache/airflow --set images.airflow.tag=master-python3.6-kubernetes -v 1 --set defaultAirflowTag=master-python3.6-kubernetes -v 1 --set config.api.auth_backend=airflow.api.auth.backend.default
2020-08-31T16:29:33.1466658Z Error: values don't meet the specifications of the schema(s) in the following chart(s):
2020-08-31T16:29:33.1467262Z airflow:
2020-08-31T16:29:33.1467809Z - fernetKey: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1468655Z - statsd.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1469091Z - registry.secretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1469800Z - rbacEnabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1470663Z - elasticsearch.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1471205Z - elasticsearch.secretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1471649Z - cleanup.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1472085Z - allowPodLaunching: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1472753Z - postgresql.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1473407Z - data.metadataSecretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1473915Z - data.resultBackendSecretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1474567Z - data.metadataConnection.host: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1475291Z - data.resultBackendConnection.host: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1475828Z - pgbouncer.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1476508Z - pgbouncer.podDisruptionBudget.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1476875Z - fernetKeySecretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1477513Z - ingress.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1478153Z - ingress.web.tls.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1478797Z - ingress.flower.tls.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1479444Z - workers.persistence.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1480094Z - workers.persistence.storageClassName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1480871Z - workers.persistence.fixPermissions: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1484488Z - workers.safeToEvict: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1492524Z - workers.keda.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1499109Z - scheduler.podDisruptionBudget.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1499838Z - scheduler.airflowLocalSettings: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1500356Z - scheduler.safeToEvict: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1501174Z - dags.persistence.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1501673Z - dags.persistence.storageClassName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1502131Z - dags.persistence.existingClaim: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1502553Z - dags.gitSync.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1502934Z - networkPolicies.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1503347Z - images.flower.tag: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1503757Z - images.flower.repository: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1504173Z - webserver.webserverConfig: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1504606Z - webserver.defaultUser.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1505087Z - redis.password: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1505510Z - redis.safeToEvict: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1505936Z - redis.persistence.enabled: Invalid type. Expected: integer, given: boolean
2020-08-31T16:29:33.1506365Z - redis.persistence.storageClassName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1506783Z - redis.passwordSecretName: Invalid type. Expected: string, given: null
2020-08-31T16:29:33.1507330Z - redis.brokerURLSecretName: Invalid type. Expected: string, given: null

@ryw ryw requested a review from dimberman September 2, 2020 12:45
"config": {
"description": "Settings to go into the mounted airflow.cfg",
"type": "object",
"properties": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here similar, but a little more complex. We expect there will be a map, which has a map that contains arbitrary text strings. Map<String, Map<String, String>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I do a

helm install airflow . --namespace airflow --set defaultAirflowRepository=apache/airflow --set images.airflow.repository=apache/airflow --set images.airflow.tag=master-python3.6-kubernetes -v 1 --set defaultAirflowTag=master-python3.6-kubernetes -v 1 --set config.api.auth_backend=airflow.api.auth.backend.default --dry-run

I get the following errors:

Error: values don't meet the specifications of the schema(s) in the following chart(s):
airflow:
- config.elasticsearch_configs.max_retries: Invalid type. Expected: string, given: integer
- config.elasticsearch_configs.timeout: Invalid type. Expected: string, given: integer
- config.scheduler.run_duration: Invalid type. Expected: string, given: integer
- config.scheduler.scheduler_heartbeat_sec: Invalid type. Expected: string, given: integer
- config.scheduler.statsd_port: Invalid type. Expected: string, given: integer

And indeed we have the following values:

config:
  scheduler:
    scheduler_heartbeat_sec: 5
    statsd_port: 9125
    run_duration: 41460
  elasticsearch_configs:
    max_retries: 3
    timeout: 30

Should we allow primitive types as well (integer, number and boolean) ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. We should allow other types also.

@mik-laj
Copy link
Member

mik-laj commented Sep 3, 2020

It would be fantastic if we could generate documentation for README.md based on this, but we should do it in a separate PR. It has a lot of potential.

@flvndh
Copy link
Contributor Author

flvndh commented Sep 7, 2020

It would be fantastic if we could generate documentation for README.md based on this, but we should do it in a separate PR. It has a lot of potential.

Indeed, this is a great way to keep both in sync. Let me know if I can help with this as well :)

@flvndh flvndh marked this pull request as ready for review September 7, 2020 14:39
@flvndh
Copy link
Contributor Author

flvndh commented Sep 17, 2020

@mik-laj @dimberman Do you have any feedback on this PR ? Also, should I clean a bit the commit history ?

@mik-laj
Copy link
Member

mik-laj commented Sep 17, 2020

Also, should I clean a bit the commit history ?

We use squash and merge so it's not important to us. If you wish, you can do so, but it is not required.

I can see your branch is a bit outdated. Can you do a rebase?

@mik-laj
Copy link
Member

mik-laj commented Sep 17, 2020

@dimberman Can you look at it? This is a nice improvement for the Helm Chart.

@flvndh flvndh force-pushed the issue/10634/validate-k8s-values-schema branch from 147744a to c4bb235 Compare September 17, 2020 18:21
@mik-laj
Copy link
Member

mik-laj commented Sep 18, 2020

@schnie Can I ask for review?

@kaxil kaxil changed the title Validate values.yaml Validate Helm values.yaml Sep 21, 2020
@kaxil kaxil merged commit a449b1f into apache:master Sep 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:helm-chart Airflow Helm Chart
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants