Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HELM] - Add guide how to PgBouncer with Kubernetes Secret #42460

Merged
merged 2 commits into from
Oct 2, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions docs/helm-chart/production-guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,84 @@ If you are using PostgreSQL as your database, you will likely want to enable `Pg
Airflow can open a lot of database connections due to its distributed nature and using a connection pooler can significantly
reduce the number of open connections on the database.

Database credentials stored Values file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: yaml

pgbouncer:
enabled: true


Database credentials stored Kubernetes Secret
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The default connection string in this case will not work you need to modify accordingly

.. code-block:: bash

kubectl create secret generic mydatabase --from-literal=connection=postgresql://user:pass@pgbouncer_svc_name.deployment_namespace:6543/airflow-metadata

Two additional Kubernetes Secret required to PgBouncer able to properly work in this configuration:

``airflow-pgbouncer-stats``

.. code-block:: bash

kubectl create secret generic airflow-pgbouncer-stats --from-literal=connection=postgresql://user:pass@127.0.0.1:6543/pgbouncer?sslmode=disable

``airflow-pgbouncer-config``

.. code-block:: yaml

apiVersion: v1
kind: Secret
metadata:
name: airflow-pgbouncer-config
data:
pgbouncer.ini: dmFsdWUtMg0KDQo=
users.txt: dmFsdWUtMg0KDQo=


``pgbouncer.ini`` equal to the base64 encoded version of this text

.. code-block:: text

[databases]
airflow-metadata = host={external_database_host} dbname={external_database_dbname} port=5432 pool_size=10

[pgbouncer]
pool_mode = transaction
listen_port = 6543
listen_addr = *
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/users.txt
stats_users = postgres
ignore_startup_parameters = extra_float_digits
max_client_conn = 100
verbose = 0
log_disconnections = 0
log_connections = 0

server_tls_sslmode = prefer
server_tls_ciphers = normal

``users.txt`` equal to the base64 encoded version of this text

.. code-block:: text

"{ external_database_host }" "{ external_database_pass }"

The ``values.yaml`` should looks like this

.. code-block:: yaml

pgbouncer:
enabled: true
configSecretName: airflow-pgbouncer-config
metricsExporterSidecar:
statsSecretName: airflow-pgbouncer-stats


Depending on the size of your Airflow instance, you may want to adjust the following as well (defaults are shown):

Expand Down