Hot fixes to minimize and close idle database connections #547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This implements a threaded approach to the datastream view, which should increase performance by insert data in parallel.
Performance profiling showed that the Django models used in the view end point for 'api/data-stream' were acting as a bottle neck, and also did not allow for asynchronous support. This commit is a patch which replaces those models with direct SQL, which should be more performant and also supports multithreading.
My previous commit (7612930) replace django models with customized queries. These queries leverages the DO operation and some Postgres IF logic. While this executes fine in Postgres, it doesn't appear to function with SQLAlchemy. This commit replace those queries with more simple logic that is supported by SQLAlchemy.
|
Note that these hot fixes required us to set Gunicorn to only have 1 worker with 8 threads, to avoid the potential problems described in Connection Pooling — SQLAlchemy 1.4 Documentation. This simple work-around leaves us we plenty of room to further enhance performance by properly configuring SQLAlchemy to use Multiprocessing. |
This was referenced Jan 6, 2022
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As described in #543 (comment), the old code that inserted data from device POSTs, based on a set of convoluted Django models, was leaving open an ever increasing number of idle connections to the PostgreSQL database server. This resulted in every increasing CPU utilization that would max out in less than a week unless the database server was rebooted as seen below:
This general issue is well-described by:
This PR replaces most of the data insert Django model code with code based on SQLAchemy. When deployed to production as a hot fix at around 16:48 UTC on Jan. 5., the base CPU load decreased and has stayed low, as shown here:
Quick additional testing showed that:
This PR fixes: