PLAT-10420: proxy support and SSL verification #122

symphony-elias · 2021-02-11T12:40:34Z

Ticket

Description

Proxy support with and without basic authentication.
Enforced SSL verification with a configurable truststore (otherwise it will use the default system CA).

Checklist

Referenced a ticket in the PR title and in the corresponding section
Filled properly the description and dependencies, if any
Unit tests updated or added
Docstrings added or updated
Updated the documentation in docs folder

symphony-elias · 2021-02-11T14:54:50Z

symphony/bdk/gen/rest.py

@@ -49,8 +49,12 @@ def __init__(self, configuration, pools_size=4, maxsize=None):
            maxsize = configuration.connection_pool_maxsize

        ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH, cafile=configuration.ssl_ca_cert)


if cafile is None, it uses the system's default CA, see https://docs.python.org/3/library/ssl.html#ssl.create_default_context

symphony-elias · 2021-02-11T14:55:17Z

symphony/bdk/gen/rest.py

        if configuration.cert_file:
-            ssl_context.load_verify_locations(configuration.cert_file)
+            ssl_context.load_cert_chain(


This was to align again to the mustache template (something I mistakenly modified). Not used so far.

symphony-youri · 2021-02-11T15:18:52Z

symphony/bdk/core/config/model/bdk_ssl_config.py

-            truststore_config = config.get(TRUST_STORE)
-            self.trust_store = BdkCertificateConfig(path=truststore_config.get("path"),
-                                                    password=truststore_config.get("password"))
+            self.trust_store = BdkCertificateConfig(path=config[TRUST_STORE].get("path"))


we don't have a password anymore?

in the java we also support passing a content (instead of a path) do we want to do it here too? (at least to support the same config files?)

No need for password: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations
In Java, we don't support passing directly a content for truststores: https://github.com/SymphonyPlatformSolutions/symphony-api-client-java/blob/master/symphony-bdk-core/src/main/java/com/symphony/bdk/core/config/model/BdkSslConfig.java
I guess you referred to RSA private keys when authenticating.
And the generated Configuration object does not support passing directly the content of the PEM. So I'd prefer to open a ticket if we ever need this.

docs/configuration.md

thibauult · 2021-02-11T15:25:48Z

docs/configuration.md

+ssl:
+  trustStore:
+    path: /path/to/truststore.pem


ssl: trustStore: path: /path/to/truststore.pem password: changeit

From https://github.com/SymphonyPlatformSolutions/symphony-api-client-java/blob/master/docs/configuration.md#full-configuration-example

Indeed but a password is not needed. We directly used a file in pem format for the truststore, see : https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations

ah you are right, in Java we use a keystore and here a PEM file!

Let's perhaps add this link to the documentation (in a comment next to the full example)? saying we support pem (cafile) parameter?

Done below in the "Configuration structure" part.

In finos#122 we enable loading a custom cert store for the HTTP client. This broked using pods with valid certs because the system certs are no longer loaded. This change loads them even all the time (and we might add custom ones on top of that). Also sync the rest.mustache template used for code generation.

PLAT-10643: Load certs from system store In #122 we enable loading a custom cert store for the HTTP client. This broked using pods with valid certs because the system certs are no longer loaded. This change loads them even all the time (and we might add custom ones on top of that). Also sync the rest.mustache template used for code generation.

* PLAT-10433: Sphinx documentation (#130) * PLAT-10483 - HealthService implementation (#132) - HealthService implementation and tests - Renamed test classes from `test_name_of_class.py` to `name_of_class_test.py` * PLAT-10532: ApplicationService implementation (#133) * ApplicationService implementation * Update documentation * PLAT-10530: Presence service implementation (#134) - Presence service implementation and tests - Put the presence service test payloads in one JSON file - Added type-hint to the ServiceFactory.get_health_service function * PLAT-10530 Fixed documentation issues in links regarding the presence service (#135) This fix was added to correctly show links to the endpoint documentation when sphinx documentation is generated. * PLAT-10531 Signal service implementation (#138) * PLAT-10529 Ability to set private key and certificate content programmatically (#137) Private key and Certificate were able to only be loaded from the config file when initializing bdk. In this PR, we update bot config to be able to set private key and certificate after bdk being initialized. * PLAT-10489: Implemented automatic pagination (#139) * PLAT-10643: Load certs from system store In #122 we enable loading a custom cert store for the HTTP client. This broked using pods with valid certs because the system certs are no longer loaded. This change loads them even all the time (and we might add custom ones on top of that). Also sync the rest.mustache template used for code generation. * PLAT-10643: Unit test system certs loading * Added forgotten default value in list_all_stream_members (#144) * PLAT-10533: Added User-Agent header in requests (#141) * PLAT-10600 enforce pylint checks in PR builder (#146) Goal of this PR is to enable pylint checks in the PR builder. For now the limit to fail is set to 9.50 (max being 10.00). We can start like this and if needed we can adapt it later on, if we think that is too strict. All changes about fixture are related to this issue: https://stackoverflow.com/questions/46089480/pytest-fixtures-redefining-name-from-outer-scope-pylint * PLAT-10534: Added X-Trace-Id header to each HTTP call (#147) * PLAT-10534: Added X-Trace-Id header to each HTTP call * Replaced single quotes with double quotes * Fixed python version in GH workflow * Updated caching logic in workflows (#148) * Changed caching logic of poetry deps to be specific to the actual python version. * Updated build, push and pylint to cache pip dependencies as well * PLAT-10651: Updated generated code following role type update to string (#149) * Updated generated code using openApiGen 5.0.1 and updated role format to str * Updated unit tests and examples * PLAT-10565 Adding utils classes (#150) Adding some util classes on message processing to: Extract entities from a given incoming message (mentions, hashtags, cashtags, emojis) Extract message content from presentationML Pre-process an outgoing message by "cleaning" the text message, escaping all special characters that will violate messageML format Convert a stream id into a url safe id and viceversa * PLAT-10535: Extension App RSA authentication (#153) * PLAT-10535: First implementation of RSA ext app authentication * Added and improved docstrings * Fixed some pylint errors * Updated poetry deps * #143 Activity API (#151) * #143 Activity API (draft version) * #143 Added unit tests * #143 Added markdown doc * #143 Documented AbstractActivity class * PLAT-10709: Switch from python-jose to PyJWT (#159) Switch from using python-jose to using PyJWT * PLAT-10588: Implemented concurrent DF loop (#158) * PLAT-10588: Implemented concurrency of event handling in DF loop * Refactored datafeed loops * Updated poetry deps * PLAT-10563: FormReply activity (#161) - Base classes for a FormReply Activity. - Modification of the command activity to parse attributes in the constructor instead - Fixed related tests * PLAT-10538: Minor improvements on extension app authenticator (#163) * PLAT-10538: added allowed audience = app-id when validating jwt * Improved error handling in ExtensionAppAuthenticatorRsa.validate_jwt * Updated dependencies * Bumped version to 2.0b1 * PR builder should build 2.0-rc branch Co-authored-by: symphony-youness <76746033+symphony-youness@users.noreply.github.com> Co-authored-by: symphony-hong <65538951+symphony-hong@users.noreply.github.com> Co-authored-by: Mariacristina De Dominicis <65179248+symphony-mariacristina@users.noreply.github.com> Co-authored-by: Soufiane Aourinmouche <52406574+symphony-soufiane@users.noreply.github.com> Co-authored-by: Youri Bonnaffe <youri.bonnaffe@symphony.com> Co-authored-by: Youri Bonnaffé <63661676+symphony-youri@users.noreply.github.com> Co-authored-by: Thibault Pensec <39826516+symphony-thibault@users.noreply.github.com>

* PLAT-10433: Sphinx documentation (finos#130) * PLAT-10483 - HealthService implementation (finos#132) - HealthService implementation and tests - Renamed test classes from `test_name_of_class.py` to `name_of_class_test.py` * PLAT-10532: ApplicationService implementation (finos#133) * ApplicationService implementation * Update documentation * PLAT-10530: Presence service implementation (finos#134) - Presence service implementation and tests - Put the presence service test payloads in one JSON file - Added type-hint to the ServiceFactory.get_health_service function * PLAT-10530 Fixed documentation issues in links regarding the presence service (finos#135) This fix was added to correctly show links to the endpoint documentation when sphinx documentation is generated. * PLAT-10531 Signal service implementation (finos#138) * PLAT-10529 Ability to set private key and certificate content programmatically (finos#137) Private key and Certificate were able to only be loaded from the config file when initializing bdk. In this PR, we update bot config to be able to set private key and certificate after bdk being initialized. * PLAT-10489: Implemented automatic pagination (finos#139) * PLAT-10643: Load certs from system store In finos#122 we enable loading a custom cert store for the HTTP client. This broked using pods with valid certs because the system certs are no longer loaded. This change loads them even all the time (and we might add custom ones on top of that). Also sync the rest.mustache template used for code generation. * PLAT-10643: Unit test system certs loading * Added forgotten default value in list_all_stream_members (finos#144) * PLAT-10533: Added User-Agent header in requests (finos#141) * PLAT-10600 enforce pylint checks in PR builder (finos#146) Goal of this PR is to enable pylint checks in the PR builder. For now the limit to fail is set to 9.50 (max being 10.00). We can start like this and if needed we can adapt it later on, if we think that is too strict. All changes about fixture are related to this issue: https://stackoverflow.com/questions/46089480/pytest-fixtures-redefining-name-from-outer-scope-pylint * PLAT-10534: Added X-Trace-Id header to each HTTP call (finos#147) * PLAT-10534: Added X-Trace-Id header to each HTTP call * Replaced single quotes with double quotes * Fixed python version in GH workflow * Updated caching logic in workflows (finos#148) * Changed caching logic of poetry deps to be specific to the actual python version. * Updated build, push and pylint to cache pip dependencies as well * PLAT-10651: Updated generated code following role type update to string (finos#149) * Updated generated code using openApiGen 5.0.1 and updated role format to str * Updated unit tests and examples * PLAT-10565 Adding utils classes (finos#150) Adding some util classes on message processing to: Extract entities from a given incoming message (mentions, hashtags, cashtags, emojis) Extract message content from presentationML Pre-process an outgoing message by "cleaning" the text message, escaping all special characters that will violate messageML format Convert a stream id into a url safe id and viceversa * PLAT-10535: Extension App RSA authentication (finos#153) * PLAT-10535: First implementation of RSA ext app authentication * Added and improved docstrings * Fixed some pylint errors * Updated poetry deps * finos#143 Activity API (finos#151) * finos#143 Activity API (draft version) * finos#143 Added unit tests * finos#143 Added markdown doc * finos#143 Documented AbstractActivity class * PLAT-10709: Switch from python-jose to PyJWT (finos#159) Switch from using python-jose to using PyJWT * PLAT-10588: Implemented concurrent DF loop (finos#158) * PLAT-10588: Implemented concurrency of event handling in DF loop * Refactored datafeed loops * Updated poetry deps * PLAT-10563: FormReply activity (finos#161) - Base classes for a FormReply Activity. - Modification of the command activity to parse attributes in the constructor instead - Fixed related tests * PLAT-10538: Minor improvements on extension app authenticator (finos#163) * PLAT-10538: added allowed audience = app-id when validating jwt * Improved error handling in ExtensionAppAuthenticatorRsa.validate_jwt * Updated dependencies * Bumped version to 2.0b1 * PR builder should build 2.0-rc branch Co-authored-by: symphony-youness <76746033+symphony-youness@users.noreply.github.com> Co-authored-by: symphony-hong <65538951+symphony-hong@users.noreply.github.com> Co-authored-by: Mariacristina De Dominicis <65179248+symphony-mariacristina@users.noreply.github.com> Co-authored-by: Soufiane Aourinmouche <52406574+symphony-soufiane@users.noreply.github.com> Co-authored-by: Youri Bonnaffe <youri.bonnaffe@symphony.com> Co-authored-by: Youri Bonnaffé <63661676+symphony-youri@users.noreply.github.com> Co-authored-by: Thibault Pensec <39826516+symphony-thibault@users.noreply.github.com>

PLAT-10420: Implemented proxy support

f204d12

symphony-elias requested review from thibauult, symphony-youri, symphony-hong and a team February 11, 2021 12:40

Implemented SSL verification

fd5363f

symphony-elias commented Feb 11, 2021

View reviewed changes

symphony-elias changed the title ~~PLAT-10420: proxy support~~ PLAT-10420: proxy support and SSL verification Feb 11, 2021

symphony-youri approved these changes Feb 11, 2021

View reviewed changes

thibauult reviewed Feb 11, 2021

View reviewed changes

docs/configuration.md Show resolved Hide resolved

thibauult reviewed Feb 11, 2021

View reviewed changes

Elias Croze added 3 commits February 11, 2021 16:42

Added example of proxy definition at component level

b4dc200

Improved documentation about the trustStore part

7b0bac6

Simplified BdkSslConfig class

9710453

symphony-youri approved these changes Feb 11, 2021

View reviewed changes

symphony-elias merged commit d47f83a into finos:2.0 Feb 12, 2021

symphony-elias deleted the PLAT-10420 branch February 18, 2021 08:50

symphony-youri mentioned this pull request Mar 8, 2021

PLAT-10643: Load certs from system store #142

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PLAT-10420: proxy support and SSL verification #122

PLAT-10420: proxy support and SSL verification #122

symphony-elias commented Feb 11, 2021 •

edited

Loading

symphony-elias Feb 11, 2021

symphony-elias Feb 11, 2021 •

edited

Loading

symphony-youri Feb 11, 2021

symphony-youri Feb 11, 2021

symphony-elias Feb 11, 2021 •

edited

Loading

thibauult Feb 11, 2021

symphony-elias Feb 11, 2021 •

edited

Loading

symphony-youri Feb 11, 2021

symphony-youri Feb 11, 2021

symphony-elias Feb 11, 2021 •

edited

Loading

		@@ -49,8 +49,12 @@ def __init__(self, configuration, pools_size=4, maxsize=None):
		maxsize = configuration.connection_pool_maxsize

		ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH, cafile=configuration.ssl_ca_cert)

PLAT-10420: proxy support and SSL verification #122

PLAT-10420: proxy support and SSL verification #122

Conversation

symphony-elias commented Feb 11, 2021 • edited Loading

Ticket

Description

Checklist

symphony-elias Feb 11, 2021

Choose a reason for hiding this comment

symphony-elias Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

symphony-youri Feb 11, 2021

Choose a reason for hiding this comment

symphony-youri Feb 11, 2021

Choose a reason for hiding this comment

symphony-elias Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

thibauult Feb 11, 2021

Choose a reason for hiding this comment

symphony-elias Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

symphony-youri Feb 11, 2021

Choose a reason for hiding this comment

symphony-youri Feb 11, 2021

Choose a reason for hiding this comment

symphony-elias Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

symphony-elias commented Feb 11, 2021 •

edited

Loading

symphony-elias Feb 11, 2021 •

edited

Loading

symphony-elias Feb 11, 2021 •

edited

Loading

symphony-elias Feb 11, 2021 •

edited

Loading

symphony-elias Feb 11, 2021 •

edited

Loading