Expand parameter overrides

[sigJoe]

Which issue(s) does this change fix?

Issues: #2380 #2253
Discussions: #4591 #6377

Why is this change necessary?

It's a roundup of quality of life improvements for users of SAM CLI that need to manage multiple environments with different parameters.

How does it address the issue?

• Adds support for YAML dictionary syntax for key-value pairs (previously only supported lists of strings)
• Flattens nested lists to better support YAML anchors and aliases for list merging
• Adds support for external file references (e.g. file://MyParams.yaml or file://MyParams.toml), which can be used both in samconfig and via CLI
  • YAML and TOML work with key-value pairs as you'd expect - just don't use tables (sections) in the TOML or you're gonna have a bad time
  • JSON SHOULD just work as soon as FILE_MANAGER_MAPPER adds it as a supported option. I could have bypassed that to directly use the JsonFileManager, but I figured there's probably a reason JSON is explicitly disabled.
• Clear precedence with later parameters overwriting previous ones

What side effects does this change have?

n/a

Mandatory Checklist

PRs will only be reviewed after checklist is complete

• Add input/output type hints to new functions/methods
• (n/a) Write design document if needed (Do I need to write a design document?)
• Write/update unit tests
• Write/update integration tests
• Write/update functional tests if needed
• make pr passes
• (n/a) make update-reproducible-reqs if dependencies were changed
• Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I don't know where I'm expected to write documentation

[roger-zhangg]

Hi @sigJoe thanks for contributing, would like to hear more opinion on the change of behavior here

If someone is relying on the previous behaviour where samconfig parameter_overrides get completely discarded when specifying CLI parameter-overrides, this PR could be an issue.

[sigJoe]

Hi @sigJoe thanks for contributing, would like to hear more opinion on the change of behavior here

If someone is relying on the previous behaviour where samconfig parameter_overrides get completely discarded when specifying CLI parameter-overrides, this PR could be an issue.

Prior to this PR, specifying --parameter-overrides on the CLI will cause SAM to completely ignore the parameter_overrides section of samconfig.tom or samcomfig.yaml.

In this PR, SAM will merge both sources of overrides and build/deploy/etc with the resulting set.

I consider the new behaviour to be objectively better. It was also the basis for #2380.

However, I admit it is conceivable that somebody might rely on the way it is currently working, even though I personally don't have any use cases for it.

Edit: I didn't really think about it when I started, but I suppose it could be a good idea to make it retain the current default behaviour and only merge samconfig when someone specifies samconfig as an override (e.g. sam deploy --parameter-overrides samconfig foo=bar file://otherparams.yaml)

[sigJoe]

After thinking about it further, I removed the merging of samconfig with CLI parameter overrides as probably deserves further discussion

[sigJoe]

To recap: I changed the syntax that failed the Python3.9 tests and removed the merging of parameters that caused that integration test to fail (can re-add if necessary but should probably be its own thing).

and here's some documentation:

Parameter Overrides Syntax

samconfig.toml

Three supported parameter_overrides formats:

version = 0.1
[default.deploy.parameters]
stack_name = "sam-app"
region = "us-east-1"
parameter_overrides = [
  "file://params.toml",                  # NEW! Amazing file syntax
  "ParameterKey=foo,ParameterValue=bar", # Long form key/value pair. Unnecessarily verbose
  "a=1"                                  # Short form key/value pair
]

You can now also use parameter_overrides as a TOML table/section

version = 0.1
[default.deploy.parameters]
stack_name = "sam-app"
region = "us-east-1"

[default.deploy.parameters.parameter_overrides]
a = "1"

Multiple parameters per string are not best practice:

version = 0.1
[default.deploy.parameters]
stack_name = "sam-app"
region = "us-east-1"
parameter_overrides = [
  "a=1 b=2",                                    # Valid to bundle multiple parameters, although not as clean as putting them on separate lines
  "c=3 ParameterKey=d,ParameterValue=4",        # Invalid: Cannot mix syntax
  "d=4 file://params.toml",                     # Invalid: Still not allowed to mix syntax
  "file://params.toml file://otherparams.toml", # Invalid: Despite being the same syntax, it didn't feel necessary to support this
]

samconfig.yaml

Old format

version: 0.1
default:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-east-1
      parameter_overrides:
        - A=1
        - B=2
        - Spaces='String with spaces'
        - ListParam='a,b,c'

NEW! Dictionary format support

Looks cleaner and supports IDE syntax highlighting

version: 0.1
default:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-east-1
      parameter_overrides:
        - A: 1
        - B: 2
        - Spaces: String with spaces
        - ListParam: # Converts to "a,b,c"
          - a
          - b
          - c
        - file://params.yaml

NEW! Anchors and aliases format support

See https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#YAML-Anchors-and-Alias for more info.
The following example shows how you can re-use configuration blocks for parameter_overrides. While anchors and aliases technically did work previously, the problem was you would end up with nested lists that would not get parsed correctly. This PR flattens nested lists such that you can now do things like this:

version: 0.1
prod-a:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-east-1
      parameter_overrides: &prod_a_overrides
        - Name: prod-a
        - ParamA: 1
        - ParamB: 2

prod-b:
  deploy:
    parameters:
      stack_name: sam-app-2
      region: us-east-1
      parameter_overrides:
        - *prod_a_overrides # Pulls in above params (Name, ParamA, and ParamB)
        - Name: prod-b      # But we want this one named differently

External TOML params files

When specifying a toml file in parameter-overrides, the file accepts basic key = value syntax. See https://toml.io/en/ for more info.

StringParam = "Pam Param"
NumberParam = 42
ListParam = "a,b,c"
ListParam2 = ["a", "b", "c"] # Converts to "a,b,c"

Note there are limitations to the format such that you cannot specify nested TOML files

StringParam = "Pam Param"
NumberParam = 42
file://another_toml_file.toml # This will fail because it is not valid TOML

External YAML params files

When specifying a yaml file in parameter-overrides, you get all the same config options
Root list:

- A=1 # OK if you really must
- ParameterKey=B,ParameterValue=2 # It works, but now you're just being silly
- StringParam: String with Spaces
- NumberParam: 42
- ListParam: a,b,c
- ListParam2: # Converts to "a,b,c"
  - a
  - b
  - c
- file://other_yaml_params.yaml
- file://toml_params_because_I_can.toml

Ideal Scenario

These features unlock a lot of different use cases, but personally I would be using it something like this:

version: 0.1
prod-a:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-east-1
      parameter_overrides: file://prod-a.yaml # includes prod.yaml, which includes common.yaml

prod-b:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-west-2
      parameter_overrides: file://prod-b.yaml # includes prod.yaml, which includes common.yaml

dev:
  deploy:
    parameters:
      stack_name: sam-app
      region: us-east-1
      parameter_overrides: file://dev.yaml # includes common.yaml

[vicheey]

Thank you for your update. Pending review from team.

[sigJoe]

Sorry about that - I'm not sure how I missed those other failing tests.

While fixing them, I realized that there was another issue of the JSON schema validation. I'm not entirely satisfied I addressed it in the best way (see schema/make_schema.py) and I get the feeling I'm missing something about how it gets used, but at least the tests are passing now.