Merge branch 'release/4.9.1'

Author: GrahamDumpleton

.github/workflows/main.yml

@@ -34,7 +34,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["2.7", "3.5", "3.6", "3.7", "3.8", "3.9", "3.10-dev"]
+        python-version: ["2.7", "3.5", "3.6", "3.7", "3.8", "3.9", "3.10", "3.11-dev"]
     steps:
       - uses: "actions/checkout@v2"
       - uses: "actions/setup-python@v2"

configure.ac

@@ -1,6 +1,6 @@
 dnl  vim: set sw=4 expandtab :
 dnl
-dnl  Copyright 2007-2021 GRAHAM DUMPLETON
+dnl  Copyright 2007-2022 GRAHAM DUMPLETON
 dnl 
 dnl  Licensed under the Apache License, Version 2.0 (the "License");
 dnl  you may not use this file except in compliance with the License.

docs/conf.py

@@ -41,7 +41,7 @@
 
 # General information about the project.
 project = u'mod_wsgi'
-copyright = u'2007-2021, Graham Dumpleton'
+copyright = u'2007-2022, Graham Dumpleton'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/release-notes.rst

@@ -5,6 +5,7 @@ Release Notes
 .. toctree::
    :maxdepth: 2
 
+   release-notes/version-4.9.1
    release-notes/version-4.9.0
 
    release-notes/version-4.8.0

docs/release-notes/version-4.4.9.rst

@@ -78,11 +78,11 @@ The header names which are accepted for specifying the target host are
 to override the ``HTTP_HOST`` value in the WSGI ``environ`` dictionary.
 
 The sole header name accepted for specifying the front end proxy server
-name is ``X-Fowarded-Server``. When found, the value will be used to
+name is ``X-Forwarded-Server``. When found, the value will be used to
 override the ``SERVER_NAME`` value in the WSGI ``environ`` dictionary.
 
 The sole header name accepted for specifying the front end proxy server
-port is ``X-Fowarded-Port``. When found, the value will be used to
+port is ``X-Forwarded-Port``. When found, the value will be used to
 override the ``SERVER_PORT`` value in the WSGI ``environ`` dictionary.
 
 The header names accepted for specifying the client IP address are

docs/release-notes/version-4.9.1.rst

@@ -0,0 +1,99 @@
+=============
+Version 4.9.1
+=============
+
+Version 4.9.1 of mod_wsgi can be obtained from:
+
+  https://codeload.github.com/GrahamDumpleton/mod_wsgi/tar.gz/4.9.1
+
+Bugs Fixed
+----------
+
+* When using ``--enable-debugger`` of mod_wsgi-express to enable Pdb, it was
+  failing due to prior changes to run Apache in a sub processes to avoid Apache
+  being shutdown when the window size changed. This was because standard input
+  was being detached from Apache and so it was not possible to interact with
+  Pdb. Now when ``--enable-debugger`` is used, or any feature which uses
+  ``--debug-mode``, Apache will not be run in a sub process so that you can
+  still use standard input to interact with the process if needed. This does
+  mean that a window size change event will again cause Apache to shutdown in
+  these cases though.
+
+* Update code so compiles on Python 3.11. Python 3.11 makes structures for
+  Python frame objects opaque and requires functions to access struct members.
+
+Features Changed
+----------------
+
+* Historically when a process was being shutdown, mod_wsgi would do its best to
+  destroy any Python sub interpreters as well as the main Python interpreter.
+  This was done in case applications attempted to run any actions on process
+  shutdown via ``atexit`` registered callbacks or other means.
+
+  Because of changes in Python 3.9, and possibly because mod_wsgi makes use of
+  externally created C threads to handle requests, and not Python native
+  threads, there is now a suspiscion that attempting to delete Python sub
+  interpreters can hang. It is believed this may relate to Python core now
+  expecting all Python thread state objects to have been deleted before the
+  Python sub interpreter can be destroyed. If they aren't then Python core
+  code can block indefinitely. If the issue isn't the externally created C
+  threads that mod_wsgi uses, it might instead be arising as a problem when a
+  hosted WSGI application creates its own background threads but they are
+  still running when the attempt is made to destroy the sub interpreter.
+
+  In the case of using daemon mode the result is that processes can hang on
+  shutdown, but will still at least be deleted after 5 seconds due to how
+  Apache process management will forcibly kill managed processes after 5
+  seconds if they do not exit cleanly themselves. In other words the issue
+  may not be noticed.
+
+  For embedded mode however, the Apache child process can hang around
+  indefinitely, possibly only being deleted if some higher level system
+  application manager such as systemd is able to detect the problem and
+  forcibly deleted the hung process.
+
+  Although mod_wsgi always attempts to ensure that the externally created C
+  threads are not still handling HTTP requests and thus not active prior to
+  destroying the Python interpreter, it is impossible to guarantee this.
+  Similarly, there is no way to guarantee that background threads created by a
+  WSGI application aren't still running. As such, it isn't possible to safely
+  attempt to delete the Python thread state objects before deleting the Python
+  sub interpreter.
+
+  Because of this uncertainty mod_wsgi now provides a way to disable the attempt
+  to destroy the Python sub interpreters or the main Python interpreter when the
+  process is being shutdown. This will though mean that ``atexit`` registered
+  callbacks will not be called if this option is enabled. It is therefore
+  important that you use mod_wsgi's own mechanism of being notified when a
+  process is being shutdown to perform any special actions.
+
+  ::
+
+    import mod_wsgi
+
+    def shutdown_handler(event, **kwargs):
+      print('SHUTDOWN-HANDLER', event, kwargs)
+
+    mod_wsgi.subscribe_shutdown(shutdown_handler)
+  
+  Use of this shutdown notification was necessary anyway to reliably attempt
+  to stop background threads created by the WSGI application since ``atexit``
+  registered callbacks are not called by Python core until after it thinks all
+  threads have been stopped. In other words, ``atexit`` register callbacks
+  couldn't be used to reliably stop background threads. Thus use of the
+  mod_wsgi mechanism for performing actions on process shutdown is the
+  preferred way.
+
+  Overall it is expected that the majority of users will not notice this
+  change as it is very rare to see WSGI applications want to perform special
+  actions on process shutdown. If you are affected, you should use mod_wsgi's
+  mechanism to perform special actions on process shutdown.
+
+  If you need to enable this mode whereby no attempt is made to destroy the
+  Python interpreter (including sub interpreters) on process shutdown, you can
+  add at global scope in the Apache configuration::
+
+    WSGIDestroyInterpreter Off
+
+  If you are using mod_wsgi-express, you can instead supply the command line
+  option ``--orphan-interpreter``.

setup.py

@@ -448,6 +448,9 @@ def _version():
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
         'Topic :: Internet :: WWW/HTTP :: WSGI',
         'Topic :: Internet :: WWW/HTTP :: WSGI :: Server'
     ],

src/server/__init__.py

@@ -353,6 +353,13 @@ def find_mimetypes():
 MaxConnectionsPerChild %(maximum_requests)s
 </IfDefine>
 
+<IfDefine ORPHAN_INTERPRETER>
+WSGIDestroyInterpreter Off
+</IfDefine>
+<IfDefine !ORPHAN_INTERPRETER>
+WSGIDestroyInterpreter On
+</IfDefine>
+
 <IfDefine !ONE_PROCESS>
 <IfDefine !EMBEDDED_MODE>
 WSGIRestrictEmbedded On
@@ -2692,6 +2699,10 @@ def add_option(platforms, *args, **kwargs):
         help='Specify the name of a separate log file to be used for '
         'the managed service.')
 
+add_option('all', '--orphan-interpreter', action='store_true',
+        default=False, help='Flag indicating whether should skip over '
+        'destroying the Python interpreter on process shutdown.')
+
 add_option('unix', '--embedded-mode', action='store_true', default=False,
         help='Flag indicating whether to run in embedded mode rather '
         'than the default daemon mode. Numerous daemon mode specific '
@@ -3348,6 +3359,9 @@ def _cmd_setup_server(command, args, options):
     else:
         options['https_url'] = None
 
+    if options['orphan_interpreter']:
+        options['httpd_arguments_list'].append('-DORPHAN_INTERPRETER')
+
     if options['embedded_mode']:
         options['httpd_arguments_list'].append('-DEMBEDDED_MODE')
         options['disable_reloading'] = True
@@ -3662,7 +3676,7 @@ def cmd_start_server(params):
     else:
         executable = posixpath.join(config['server_root'], 'apachectl')
 
-        if sys.stdout.isatty():
+        if sys.stdout.isatty() and not config['debug_mode']:
             process = None
 
             def handler(signum, frame):

src/server/management/commands/runmodwsgi.py

@@ -140,7 +140,7 @@ def handle(self, *args, **options):
         executable = os.path.join(options['server_root'], 'apachectl')
         name = executable.ljust(len(options['process_name']))
 
-        if sys.stdout.isatty():
+        if sys.stdout.isatty() and not options['debug_mode']:
             process = None
 
             def handler(signum, frame):

src/server/mod_wsgi.c

@@ -1,7 +1,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -4306,6 +4306,11 @@ static apr_status_t wsgi_python_child_cleanup(void *data)
     wsgi_publish_process_stopping(wsgi_shutdown_reason);
 #endif
 
+    /* Skip destruction of Python interpreter. */
+
+    if (wsgi_server_config->destroy_interpreter == 0)
+        return APR_SUCCESS;
+
     /* In a multithreaded MPM must protect table. */
 
 #if APR_HAS_THREADS
@@ -5043,6 +5048,28 @@ static const char *wsgi_set_python_hash_seed(cmd_parms *cmd, void *mconfig,
     return NULL;
 }
 
+static const char *wsgi_set_destroy_interpreter(cmd_parms *cmd, void *mconfig,
+                                                const char *f)
+{
+    const char *error = NULL;
+    WSGIServerConfig *sconfig = NULL;
+
+    error = ap_check_cmd_context(cmd, GLOBAL_ONLY);
+    if (error != NULL)
+        return error;
+
+    sconfig = ap_get_module_config(cmd->server->module_config, &wsgi_module);
+
+    if (strcasecmp(f, "Off") == 0)
+        sconfig->destroy_interpreter = 0;
+    else if (strcasecmp(f, "On") == 0)
+        sconfig->destroy_interpreter = 1;
+    else
+        return "WSGIDestroyInterpreter must be one of: Off | On";
+
+    return NULL;
+}
+
 static const char *wsgi_set_restrict_embedded(cmd_parms *cmd, void *mconfig,
                                               const char *f)
 {
@@ -9521,17 +9548,26 @@ static void wsgi_log_stack_traces(void)
                     const char *filename = NULL;
                     const char *name = NULL;
 
+#if PY_MAJOR_VERSION > 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 9)
+                    lineno = PyFrame_GetLineNumber(current);
+#else
                     if (current->f_trace) {
                         lineno = current->f_lineno;
                     }
                     else {
                         lineno = PyCode_Addr2Line(current->f_code,
                                                   current->f_lasti);
                     }
+#endif
 
 #if PY_MAJOR_VERSION >= 3
+#if PY_MAJOR_VERSION > 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 9)
+                    filename = PyUnicode_AsUTF8(PyFrame_GetCode(current)->co_filename);
+                    name = PyUnicode_AsUTF8(PyFrame_GetCode(current)->co_name);
+#else
                     filename = PyUnicode_AsUTF8(current->f_code->co_filename);
                     name = PyUnicode_AsUTF8(current->f_code->co_name);
+#endif
 #else
                     filename = PyString_AsString(current->f_code->co_filename);
                     name = PyString_AsString(current->f_code->co_name);
@@ -9544,7 +9580,11 @@ static void wsgi_log_stack_traces(void)
                                 getpid(), thread_id, filename, lineno, name);
                     }
                     else {
+#if PY_MAJOR_VERSION > 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 9)
+                        if (PyFrame_GetBack(current)) {
+#else
                         if (current->f_back) {
+#endif
                             ap_log_error(APLOG_MARK, APLOG_INFO, 0, wsgi_server,
                                     "mod_wsgi (pid=%d): called from file "
                                     "\"%s\", line %d, in %s,", getpid(),
@@ -9558,7 +9598,11 @@ static void wsgi_log_stack_traces(void)
                         }
                     }
 
+#if PY_MAJOR_VERSION > 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 9)
+                    current = PyFrame_GetBack(current);
+#else
                     current = current->f_back;
+#endif
                 }
             }
         }
@@ -10011,7 +10055,7 @@ static int wsgi_start_process(apr_pool_t *p, WSGIDaemonProcess *daemon)
 
             if (status != APR_SUCCESS) {
                 ap_log_error(APLOG_MARK, APLOG_CRIT, 0, wsgi_server,
-                             "mod_wsgi (pid=%d): Couldn't intialise accept "
+                             "mod_wsgi (pid=%d): Couldn't initialise accept "
                              "mutex in daemon process '%s'.",
                              getpid(), daemon->group->mutex_path);
 
@@ -16224,6 +16268,9 @@ static const command_rec wsgi_commands[] =
     AP_INIT_TAKE1("WSGIPythonHashSeed", wsgi_set_python_hash_seed,
         NULL, RSRC_CONF, "Python hash seed."),
 
+    AP_INIT_TAKE1("WSGIDestroyInterpreter", wsgi_set_destroy_interpreter,
+        NULL, RSRC_CONF, "Enable/Disable destruction of Python interpreter."),
+
 #if defined(MOD_WSGI_WITH_DAEMONS)
     AP_INIT_TAKE1("WSGIRestrictEmbedded", wsgi_set_restrict_embedded,
         NULL, RSRC_CONF, "Enable/Disable use of embedded mode."),

src/server/wsgi_apache.c

@@ -1,7 +1,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_apache.h

@@ -4,7 +4,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_buckets.c

@@ -1,7 +1,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_buckets.h

@@ -4,7 +4,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_convert.c

@@ -1,7 +1,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_convert.h

@@ -4,7 +4,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

src/server/wsgi_daemon.c

@@ -1,7 +1,7 @@
 /* ------------------------------------------------------------------------- */
 
 /*
- * Copyright 2007-2021 GRAHAM DUMPLETON
+ * Copyright 2007-2022 GRAHAM DUMPLETON
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.