[Bug]: Merged configuration file always has the optional properties with their default values if they are not specified in the overridden file #1737
Comments
2 tasks
2 tasks
2 tasks
Aniruddh25
added a commit
that referenced
this issue
Oct 20, 2023
## Why make this change? - As per the JSON config schema, not all properties in the config file are mandatory. - With #1402, in versions 0.8.49-0.8.52 those properties that are not explicitly set, were initialized with defaults before writing to the file system. - However, config files generated using version 0.7.6 or lesser, may not have these properties initialized to their defaults when writing to the file system. Using config files generated with <= 0.7.6, to start engine with 0.8.49 would therefore lead to null reference exceptions when the optional properties are dereferenced. - With PR #1684 the optional properties were initialized with their default values to avoid the null reference, any config file that is created or modified using the CLI results in a fully expanded file. This is not a problem when a single config is used but leads to undesired behavior when using the merge configuration file feature. A fully expanded higher precedence config file when merged with a base config file will override the values set for the optional properties in the base file with their default values if those properties were not explicitly defined in the higher precedence config file to begin with. In order to retain the base config file values for the optional properties, the higher precedence file would have to define every single optional value exactly the same as in the base file if they desire those to not be overridden. That defeats the purpose of providing the higher precedence file which should ideally only have those properties that need to be overriden. For more details with examples, please refer to #1737. - This PR fixes this problem by allowing optional properties to be null and ensures there are no null reference exceptions as well when starting engine with configs that don't have optional properties. ## What is this change? - Modify object model constructors to accept nullable arguments for optional properties - represented as fields in the record types. The scalar optional properties have not been made nullable yet, will have a follow up PR for those. - All references to optional properties check for nulls - During merge of config files, null + non-null values will pick up non-null values. After the merge, if the value is still null, they are considered as default values. To easily get default values, added some properties to `RuntimeConfig`. - Default value of Host.Mode is `Production`. Keep it that way. Default of GraphQL.AllowIntrospection is true currently - will have another PR to determine it based on host mode. ## How was this tested? - Removed optional properties from config and ensured engine could be started without them. - [X] Unit Tests for deserialization and serialization of nullable optional properties see `RuntimeConfigLoaderJsonDeserializerTests` - [X] In the merge configuration tests, in `TestHelper`, modify base config to contain a non-default value, env based config to not specify that value and confirm the merged config file has the base-config value. ## Sample Request(s) - Before fix, remove `runtime` section from config, run dab start with 0.8.49 leads to NullReferenceException. - After the fix, starting engine is successful even when `runtime` section is absent in the config. // IN THE BASE config file MISSING = use default { empty } = use default // IN THE OVERRIDDEN config file MISSING = use base { empty } = use base Examples: // all subproperties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { empty } merged: "user": { "first": "jerry", "last": "nixon" } // some sub properties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { "first": "jerry" } merged: "user": { "first": "jerry", "last": "nixon" } // parent object property is missing altogether base: "user": { "first": "jerry", "last": "nixon" } override: "user": MISSING (not specified in override) merged: "user": { "first": "jerry", "last": "nixon" } ## TO DO for 0.10-rc - A followup PR to make scalar optional properties nullable too. - A followup PR that makes specifying an explicit NULL imply Default values
Aniruddh25
added a commit
that referenced
this issue
Oct 20, 2023
## Why make this change? - As per the JSON config schema, not all properties in the config file are mandatory. - With #1402, in versions 0.8.49-0.8.52 those properties that are not explicitly set, were initialized with defaults before writing to the file system. - However, config files generated using version 0.7.6 or lesser, may not have these properties initialized to their defaults when writing to the file system. Using config files generated with <= 0.7.6, to start engine with 0.8.49 would therefore lead to null reference exceptions when the optional properties are dereferenced. - With PR #1684 the optional properties were initialized with their default values to avoid the null reference, any config file that is created or modified using the CLI results in a fully expanded file. This is not a problem when a single config is used but leads to undesired behavior when using the merge configuration file feature. A fully expanded higher precedence config file when merged with a base config file will override the values set for the optional properties in the base file with their default values if those properties were not explicitly defined in the higher precedence config file to begin with. In order to retain the base config file values for the optional properties, the higher precedence file would have to define every single optional value exactly the same as in the base file if they desire those to not be overridden. That defeats the purpose of providing the higher precedence file which should ideally only have those properties that need to be overriden. For more details with examples, please refer to #1737. - This PR fixes this problem by allowing optional properties to be null and ensures there are no null reference exceptions as well when starting engine with configs that don't have optional properties. ## What is this change? - Modify object model constructors to accept nullable arguments for optional properties - represented as fields in the record types. The scalar optional properties have not been made nullable yet, will have a follow up PR for those. - All references to optional properties check for nulls - During merge of config files, null + non-null values will pick up non-null values. After the merge, if the value is still null, they are considered as default values. To easily get default values, added some properties to `RuntimeConfig`. - Default value of Host.Mode is `Production`. Keep it that way. Default of GraphQL.AllowIntrospection is true currently - will have another PR to determine it based on host mode. ## How was this tested? - Removed optional properties from config and ensured engine could be started without them. - [X] Unit Tests for deserialization and serialization of nullable optional properties see `RuntimeConfigLoaderJsonDeserializerTests` - [X] In the merge configuration tests, in `TestHelper`, modify base config to contain a non-default value, env based config to not specify that value and confirm the merged config file has the base-config value. ## Sample Request(s) - Before fix, remove `runtime` section from config, run dab start with 0.8.49 leads to NullReferenceException. - After the fix, starting engine is successful even when `runtime` section is absent in the config. // IN THE BASE config file MISSING = use default { empty } = use default // IN THE OVERRIDDEN config file MISSING = use base { empty } = use base Examples: // all subproperties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { empty } merged: "user": { "first": "jerry", "last": "nixon" } // some sub properties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { "first": "jerry" } merged: "user": { "first": "jerry", "last": "nixon" } // parent object property is missing altogether base: "user": { "first": "jerry", "last": "nixon" } override: "user": MISSING (not specified in override) merged: "user": { "first": "jerry", "last": "nixon" } ## TO DO for 0.10-rc - A followup PR to make scalar optional properties nullable too. - A followup PR that makes specifying an explicit NULL imply Default values
seantleonard
pushed a commit
that referenced
this issue
Oct 21, 2023
## Why make this change? - As per the JSON config schema, not all properties in the config file are mandatory. - With #1402, in versions 0.8.49-0.8.52 those properties that are not explicitly set, were initialized with defaults before writing to the file system. - However, config files generated using version 0.7.6 or lesser, may not have these properties initialized to their defaults when writing to the file system. Using config files generated with <= 0.7.6, to start engine with 0.8.49 would therefore lead to null reference exceptions when the optional properties are dereferenced. - With PR #1684 the optional properties were initialized with their default values to avoid the null reference, any config file that is created or modified using the CLI results in a fully expanded file. This is not a problem when a single config is used but leads to undesired behavior when using the merge configuration file feature. A fully expanded higher precedence config file when merged with a base config file will override the values set for the optional properties in the base file with their default values if those properties were not explicitly defined in the higher precedence config file to begin with. In order to retain the base config file values for the optional properties, the higher precedence file would have to define every single optional value exactly the same as in the base file if they desire those to not be overridden. That defeats the purpose of providing the higher precedence file which should ideally only have those properties that need to be overriden. For more details with examples, please refer to #1737. - This PR fixes this problem by allowing optional properties to be null and ensures there are no null reference exceptions as well when starting engine with configs that don't have optional properties. ## What is this change? - Modify object model constructors to accept nullable arguments for optional properties - represented as fields in the record types. The scalar optional properties have not been made nullable yet, will have a follow up PR for those. - All references to optional properties check for nulls - During merge of config files, null + non-null values will pick up non-null values. After the merge, if the value is still null, they are considered as default values. To easily get default values, added some properties to `RuntimeConfig`. - Default value of Host.Mode is `Production`. Keep it that way. Default of GraphQL.AllowIntrospection is true currently - will have another PR to determine it based on host mode. ## How was this tested? - Removed optional properties from config and ensured engine could be started without them. - [X] Unit Tests for deserialization and serialization of nullable optional properties see `RuntimeConfigLoaderJsonDeserializerTests` - [X] In the merge configuration tests, in `TestHelper`, modify base config to contain a non-default value, env based config to not specify that value and confirm the merged config file has the base-config value. ## Sample Request(s) - Before fix, remove `runtime` section from config, run dab start with 0.8.49 leads to NullReferenceException. - After the fix, starting engine is successful even when `runtime` section is absent in the config. // IN THE BASE config file MISSING = use default { empty } = use default // IN THE OVERRIDDEN config file MISSING = use base { empty } = use base Examples: // all subproperties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { empty } merged: "user": { "first": "jerry", "last": "nixon" } // some sub properties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { "first": "jerry" } merged: "user": { "first": "jerry", "last": "nixon" } // parent object property is missing altogether base: "user": { "first": "jerry", "last": "nixon" } override: "user": MISSING (not specified in override) merged: "user": { "first": "jerry", "last": "nixon" } ## TO DO for 0.10-rc - A followup PR to make scalar optional properties nullable too. - A followup PR that makes specifying an explicit NULL imply Default values
2 tasks
seantleonard
pushed a commit
that referenced
this issue
Oct 23, 2023
## Why make this change? - As per the JSON config schema, not all properties in the config file are mandatory. - With #1402, in versions 0.8.49-0.8.52 those properties that are not explicitly set, were initialized with defaults before writing to the file system. - However, config files generated using version 0.7.6 or lesser, may not have these properties initialized to their defaults when writing to the file system. Using config files generated with <= 0.7.6, to start engine with 0.8.49 would therefore lead to null reference exceptions when the optional properties are dereferenced. - With PR #1684 the optional properties were initialized with their default values to avoid the null reference, any config file that is created or modified using the CLI results in a fully expanded file. This is not a problem when a single config is used but leads to undesired behavior when using the merge configuration file feature. A fully expanded higher precedence config file when merged with a base config file will override the values set for the optional properties in the base file with their default values if those properties were not explicitly defined in the higher precedence config file to begin with. In order to retain the base config file values for the optional properties, the higher precedence file would have to define every single optional value exactly the same as in the base file if they desire those to not be overridden. That defeats the purpose of providing the higher precedence file which should ideally only have those properties that need to be overriden. For more details with examples, please refer to #1737. - This PR fixes this problem by allowing optional properties to be null and ensures there are no null reference exceptions as well when starting engine with configs that don't have optional properties. ## What is this change? - Modify object model constructors to accept nullable arguments for optional properties - represented as fields in the record types. The scalar optional properties have not been made nullable yet, will have a follow up PR for those. - All references to optional properties check for nulls - During merge of config files, null + non-null values will pick up non-null values. After the merge, if the value is still null, they are considered as default values. To easily get default values, added some properties to `RuntimeConfig`. - Default value of Host.Mode is `Production`. Keep it that way. Default of GraphQL.AllowIntrospection is true currently - will have another PR to determine it based on host mode. ## How was this tested? - Removed optional properties from config and ensured engine could be started without them. - [X] Unit Tests for deserialization and serialization of nullable optional properties see `RuntimeConfigLoaderJsonDeserializerTests` - [X] In the merge configuration tests, in `TestHelper`, modify base config to contain a non-default value, env based config to not specify that value and confirm the merged config file has the base-config value. ## Sample Request(s) - Before fix, remove `runtime` section from config, run dab start with 0.8.49 leads to NullReferenceException. - After the fix, starting engine is successful even when `runtime` section is absent in the config. // IN THE BASE config file MISSING = use default { empty } = use default // IN THE OVERRIDDEN config file MISSING = use base { empty } = use base Examples: // all subproperties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { empty } merged: "user": { "first": "jerry", "last": "nixon" } // some sub properties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { "first": "jerry" } merged: "user": { "first": "jerry", "last": "nixon" } // parent object property is missing altogether base: "user": { "first": "jerry", "last": "nixon" } override: "user": MISSING (not specified in override) merged: "user": { "first": "jerry", "last": "nixon" } ## TO DO for 0.10-rc - A followup PR to make scalar optional properties nullable too. - A followup PR that makes specifying an explicit NULL imply Default values
2 tasks
seantleonard
added a commit
that referenced
this issue
Oct 24, 2023
Cherry pick #1823 to 0.9 branch. ## Why make this change? - As per the JSON config schema, not all properties in the config file are mandatory. - With #1402, in versions 0.8.49-0.8.52 those properties that are not explicitly set, were initialized with defaults before writing to the file system. - However, config files generated using version 0.7.6 or lesser, may not have these properties initialized to their defaults when writing to the file system. Using config files generated with <= 0.7.6, to start engine with 0.8.49 would therefore lead to null reference exceptions when the optional properties are dereferenced. - With PR #1684 the optional properties were initialized with their default values to avoid the null reference, any config file that is created or modified using the CLI results in a fully expanded file. This is not a problem when a single config is used but leads to undesired behavior when using the merge configuration file feature. A fully expanded higher precedence config file when merged with a base config file will override the values set for the optional properties in the base file with their default values if those properties were not explicitly defined in the higher precedence config file to begin with. In order to retain the base config file values for the optional properties, the higher precedence file would have to define every single optional value exactly the same as in the base file if they desire those to not be overridden. That defeats the purpose of providing the higher precedence file which should ideally only have those properties that need to be overriden. For more details with examples, please refer to #1737. - This PR fixes this problem by allowing optional properties to be null and ensures there are no null reference exceptions as well when starting engine with configs that don't have optional properties. ## What is this change? - Modify object model constructors to accept nullable arguments for optional properties - represented as fields in the record types. The scalar optional properties have not been made nullable yet, will have a follow up PR for those. - All references to optional properties check for nulls - During merge of config files, null + non-null values will pick up non-null values. After the merge, if the value is still null, they are considered as default values. To easily get default values, added some properties to `RuntimeConfig`. - Default value of Host.Mode is `Production`. Keep it that way. Default of GraphQL.AllowIntrospection is true currently - will have another PR to determine it based on host mode. ## How was this tested? - Removed optional properties from config and ensured engine could be started without them. - [X] Unit Tests for deserialization and serialization of nullable optional properties see `RuntimeConfigLoaderJsonDeserializerTests` - [X] In the merge configuration tests, in `TestHelper`, modify base config to contain a non-default value, env based config to not specify that value and confirm the merged config file has the base-config value. ## Sample Request(s) - Before fix, remove `runtime` section from config, run dab start with 0.8.49 leads to NullReferenceException. - After the fix, starting engine is successful even when `runtime` section is absent in the config. // IN THE BASE config file MISSING = use default { empty } = use default // IN THE OVERRIDDEN config file MISSING = use base { empty } = use base Examples: // all subproperties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { empty } merged: "user": { "first": "jerry", "last": "nixon" } // some sub properties are missing base: "user": { "first": "jerry", "last": "nixon" } override: "user": { "first": "jerry" } merged: "user": { "first": "jerry", "last": "nixon" } // parent object property is missing altogether base: "user": { "first": "jerry", "last": "nixon" } override: "user": MISSING (not specified in override) merged: "user": { "first": "jerry", "last": "nixon" } ## TO DO for 0.10-rc - A followup PR to make scalar optional properties nullable too. - A followup PR that makes specifying an explicit NULL imply Default values --------- Co-authored-by: Aniruddh Munde <anmunde@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
Example:
dab-config.json => base config has graphql enabled = false.
dab-config.development.json => no graphql options provided explicitly
We deserialize and initialize the “graphql” options with enabled = true for dab-config.development.json
Merging the base dab-config.json + the override- dab-config.development.json would lead to this:
Even though, correct merge should have retained the base config, i.e.
enabled =false.Version
0.8.51
What database are you using?
Azure SQL
What hosting model are you using?
Local (including CLI)
Which API approach are you accessing DAB through?
REST, GraphQL
Relevant log output
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: