[PY] Use ProxyManager instead of PoolManager in remote_connection

[pkvprakash]

Use ProxyManager instead of PoolManager in remote_connection to allow using a proxy to connect to the grid.

Description

Currently urllib3.PoolManager is used in creating remote
connection to the grid. If a proxy is required to connect
to the grid, we have to use urllib3.ProxyManager instead
of urllib3.PoolManager.

The proxy settings will be taken from https_proxy or http_proxy
environment variables based on the remote grid url passed to the
RemoteConnection class. If proxy environment variable is set,
ProxyManager will be used to create the connection. Otherwise
PoolManager will be used.

Fixes #8053

Motivation and Context

This change is required to connect to the grid from behind a corporate proxy.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• I have read the contributing document.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[CLAassistant]

All committers have signed the CLA.

[pkvprakash]

@AutomatedTester can you please review this change?

[AutomatedTester]

If this should be picking this up from an environment variable should we just check that instead of adding something to the arguments?

[pkvprakash]

@AutomatedTester updated the PR to take the proxy_url from environment.

[artem-fedorov]

Hi David @AutomatedTester,
When do you plan release this fix?

[AutomatedTester]

The aim is for a new alpha this week

…

On May 25, 2020, 9:52 PM +0100, Artem Fedorov ***@***.***>, wrote: Hi David @AutomatedTester, When do you plan release this fix? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[artem-fedorov]

Hi David @AutomatedTester ,
Maybe you can help with the following issue. When I use urllib3.ProxyManager the request to my server passed successfully. When I use selenium the same request crashes with:

Traceback (most recent call last):
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 662, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 948, in _prepare_proxy
    conn.connect()
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f012571c828>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/artifacts/test_requests.py", line 35, in setUp
    self.driver = webdriver.Firefox(profile, options=options)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
    keep_alive=True)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 384, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 407, in _request
    resp = self._conn.request(method, url, body=body, headers=headers)
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/request.py", line 80, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/request.py", line 171, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/poolmanager.py", line 466, in urlopen
    return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/poolmanager.py", line 328, in urlopen
    response = conn.urlopen(method, url, **kw)
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/taurus-cloud/python/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='MY_HOST', port=443): Max retries exceeded with url: /wd/hub/session (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe7d8254978>: Failed to establish a new connection: [Errno -2] Name or service not known',)))

I use selenium 3 with the similar patch that I did in remote/remote_connection.py with urllib3.ProxyManager, but it did not helped me. Maybe you know any places where selenium adds some custom configuration to urllib3 that can cause such issue?

[AutomatedTester]

I suggest raising an issue with a script to reproduce the error after trying to use thise with the version 4.0.0.a6.post1 from Pypi.

[artem-fedorov]

Sure, we'll try the latest version 4.0.0.a6.post1

[artem-fedorov]

After update to the latest 4.0.0.a6.post1 version the issue was solved. thank you!