feat(metadata): use manifest to drive config

[williamhaley]

Jira Ticket: HP-291, HP-320

New Features

• Create configuration scripts for metadata-service

Improvements

• Move aggregate MDS configuration to our standard manifest config process

Deployment changes

• Remove USE_AGG_MDS and AGG_MDS_NAMESPACE from Gen3Secrets/g3auto/metadata/metadata.env and set those variables in a manifest: {} block in manifest.json
• Migrate the appropriate agg MDS config to the relevant manifest repo under a metadata/aggregate_config.json path
• Run gen3 kube-setup-metdata and roll the metadata service in Kubernetes

[jawadqur]

[williamhaley]

I'm trying to do one thing at a time, but in conjunction with this PR we should update the cdis-manifest entries as needed add an aggregate_config.json file per-commons. I'll work on the automated process/init process for importing data automatically on a roll or whatever, but the immediate change would be...

kubectl exec $(gen3 pod metadata) -- python /src/src/mds/populate.py --config /aggregate_config.json --hostname esproxy-service --port 9200

[jawadqur]

Can we have this in an init-container instead? The init-container can run this before starting the service.

[jawadqur]

If there's no config then init container will exit 0

[williamhaley]

Yeah. I think you're right. I was going to wait to do https://ctds-planx.atlassian.net/browse/HP-287, but I think I may as well do this while I'm updating the rest of it

[williamhaley]

18db45d

[williamhaley]

Took a little longer (a bit of bash trial and error) but see latest @jawadqur. Updates were made to add an init container like you suggested. Seems to be working properly to exit 0 if not configured properly or run the import as needed so that a gen3 roll metadata is all that's needed to do an import

[williamhaley]

Taking functionality for https://ctds-planx.atlassian.net/browse/HP-287 out of this PR. This work is more complex than should be taken on here and impacting the scope. We can't easily run the agg MDS import on post init or in an init container because an agg MDS may refer to its own non-agg MDS as a datasource. Chicken and the egg. This probably best belongs in application code and not in k8s config

[jawadqur]

Taking functionality for https://ctds-planx.atlassian.net/browse/HP-287 out of this PR. This work is more complex than should be taken on here and impacting the scope. We can't easily run the agg MDS import on post init or in an init container because an agg MDS may refer to its own non-agg MDS as a datasource. Chicken and the egg. This probably best belongs in application code and not in k8s config

Makes sense, let's revisit later.

[williamhaley]

@themarcelor @mfshao regarding our offline conversation around backwards compatibility, I've found that using optional volumes for k8s does seem to work as expected in an array of situations and I believe this PR now covers the current state and future state

Testing

The various configuration mechanisms were toggled/deleted while I did a gen3 roll all to validate the health of metadata.

All scenarios worked without issue.

The metadata.env changes that are now deprecated were like so...

USE_AGG_MDS=true
AGG_MDS_NAMESPACE=my_namespace

g3auto secrets and configs would be synced like so (using the appropriate branch) ...

MANIFEST_BRANCH=master gen3 kube-setup-metadata

Configs and secrets were cherry-picked for deletion and verified like so ...

g3kubectl delete secret metadata-config ; g3kubectl delete configmap manifest-metadata
gen3 secrets decode metadata-g3auto ; gen3 secrets decode metadata-config ; g3kubectl get configmap manifest-metadata -o json
gen3 secrets sync

Service status was verified like so ...

kubectl describe pod $(gen3 pod metadata)

+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| config in metadata.env | metadata-config | manifest-metadata | manifest MDS config  | manifest agg mds conf | cloud automation | mds image        |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | non-existent    | non-existent      | non-existent         | not defined           | master           | master           |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds enabled        | non-existent    | non-existent      | non-existent         | not defined           | master           | feat_mds-adapter |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | non-existent    | non-existent      | non-existent         | not defined           | feat/metadata    | master           |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | non-existent    | auto-populated    | agg mds disabled     | not defined           | feat/metadata    | master           |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | non-existent    | auto-populated    | agg mds enabled      | not defined           | feat/metadata    | master           |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | non-existant    | auto-populated    | agg mds enabled      | not defined           | feat/metadata    | feat_mds-adapter |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | auto-populated  | auto-populated    | agg mds enabled      | defined               | feat/metadata    | feat_mds-adapter |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+
| agg mds disabled       | deleted         | deleted           | agg mds enabled      | defined               | feat/metadata    | feat_mds-adapter |
+------------------------+-----------------+-------------------+----------------------+-----------------------+------------------+------------------+