mysql module, "cannot use a string pattern on a bytes-like object"

[lsh-0]

Description of Issue

state mysql_user.present is raising a Python TypeError with the message "cannot use a string pattern on a bytes-like object" in Salt 2018.3.4 (no sensitive information).

Possibly related to #45026

Setup

our state is very simple and looks like this (in context):

mysql-root-user:
    mysql_user.present:
        - name: {{ pillar.org.db_root.username }}
        - password: {{ pillar.org.db_root.password }}
        - host: localhost
        - require:
            - mysql-server

and the (dummy, public) pillar data looks like this (in context):

org:
   ...
    db_root:
        username: root
        password: root

Steps to Reproduce Issue

Running highstate with -l debug gives us this:

[INFO    ] Executing state mysql_user.present for [root]
[DEBUG   ] Doing query: SELECT VERSION()
[DEBUG   ] Doing query: SELECT column_name from information_schema.COLUMNS WHERE table_schema=%(schema)s and table_name=%(table)s and column_name=%(column)s args: {'column': 'Password', 'schema': 'mysql', 'table': 'user'} 
[DEBUG   ] Doing query: SELECT User,Host FROM mysql.user WHERE User = %(user)s AND Host = %(host)s AND authentication_string = PASSWORD(%(password)s) args: {'password': 'root', 'host': '%', 'user': 'root'} 
[DEBUG   ] An exception occurred in this state: cannot use a string pattern on a bytes-like object
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 1919, in call
    **cdata['kwargs'])
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1918, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/mysql_user.py", line 142, in present
    **connection_args):
  File "/usr/lib/python3/dist-packages/salt/modules/mysql.py", line 1315, in user_exists
    _execute(cur, qry, args)
  File "/usr/lib/python3/dist-packages/salt/modules/mysql.py", line 609, in _execute
    return cur.execute(qry, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 228, in execute
    if not self._defer_warnings: self._warning_check()
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 127, in _warning_check
    warn(w[-1], self.Warning, 3)
TypeError: cannot use a string pattern on a bytes-like object

Versions Report

(masterless)

$ salt-call --versions-report
Salt Version:
           Salt: 2018.3.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 (default, Oct  8 2019, 13:06:37)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[lsh-0]

I can't replicate this on Ubuntu 18.04

$ salt-call --versions-report
Salt Version:
           Salt: 2018.3.4
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.3.10
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Nov  7 2019, 10:44:02)
   python-gnupg: 0.4.1
         PyYAML: 5.2
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-72-generic
         system: Linux
        version: Ubuntu 18.04 bionic
 

[eykd]

We encountered this error as well. After some debugging I tracked it down to version 1.3.7 of the mysqlclient package:

$ salt-call --versions-report|grep mysql-python
   mysql-python: 1.3.7

MySQLdb was not decoding bytes warnings generated by a query, instead passing them directly to warnings.warn(). Because salt uses warnings.filterwarnings(), the warnings subsystem was trying to apply regex patterns to the bytes, resulting in the error.

After upgrading to the latest 1.3 series:

python3 -m pip install --upgrade 'mysqlclient<1.4'

We can now run mysql states without issue.

$ salt-call --versions-report|grep mysql-python
   mysql-python: 1.3.14

[eykd]

An addendum: for us on Ubuntu 16.04, the above fixed the problem locally using salt-call but we were still seeing the problem when applying states to salt-minion from the salt-master. Removing the python-mysqldb apt package appears to have addressed this problem.

sudo apt remove python-mysqldb

I don't know why salt-minion was getting the old version (1.3.7) while salt-call was getting the new (1.3.14).

[sagetherage]

This was corrected: #50823 for Salt v2018.3.x - what version of Salt are you still seeing this @eykd ?

[eykd]

@sagetherage If I'm reading our upgrade history right, we were on 2019.2.5 at the time of my above comments. We've since made the jump to the 3001 and 3002 series. I haven't heard of this problem continuing since implementing the mitigation I described.

[sagetherage]

Thank you @eykd then I am going to close this issue since v2018.3.x and v2019.2.5 are no longer supported versions of salt - but mention me if you don't agree or you can open a new issue. thank you!