Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log full vacuum info #26633

Merged
merged 9 commits into from
May 30, 2023
Merged

Conversation

rodireich
Copy link
Contributor

What

This change logs before syncing the result of 2 queries that will allow us to get information about whether a full vacuum of the table is currently running or ran.

How

Performs two queries showing information about currently running full vacuum and also system information that allows gauging whether a full sync ran in between source-postgres attempts (relation filename).
See tech spec for more information.

🚨 User Impact 🚨

This is only adding logging.

@rodireich rodireich requested a review from a team as a code owner May 25, 2023 23:02
@rodireich rodireich linked an issue May 25, 2023 that may be closed by this pull request
@github-actions
Copy link
Contributor

github-actions bot commented May 25, 2023

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
  • You've updated the connector's metadata.yaml file (new!)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@rodireich
Copy link
Contributor Author

rodireich commented May 25, 2023

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5085364028
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5085364028
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
=================== 68 passed, 5 skipped in 82.58s (0:01:22) ===================

@rodireich
Copy link
Contributor Author

rodireich commented May 25, 2023

/connector-performance connector=connectors/source-postgres

Note: The following dataset= values are supported: 1m(default), 10m, 20m,

bottleneck_stream1, bottleneck_stream_randomseed. For destinations only: you can also use stream-numbers=N to simulate N number of parallel streams. Additionally,sync-mode=incrementalis supported for destinations. For example:dataset=1m stream-numbers=2 sync-mode=incremental`

🏃 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5085366836.

Performance test Result:

{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.PerformanceTest(runTest):108 total secs: 42.056. total MB read: 524.64319896698, rps: 52571.83279436942, throughput: 12.47487157520877

@github-actions
Copy link
Contributor

github-actions bot commented May 25, 2023

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (3)

Connector Version Changelog Publish
source-alloydb 2.0.28
source-alloydb-strict-encrypt 2.0.28 🔵
(ignored)
🔵
(ignored)
source-postgres-strict-encrypt 2.0.31 🔵
(ignored)
🔵
(ignored)
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category Status Actionable Item
Version
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.

doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.

changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish
not in seed
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

@rodireich rodireich requested a review from akashkulk May 26, 2023 22:51
@rodireich rodireich requested a review from akashkulk May 30, 2023 18:03
@rodireich rodireich enabled auto-merge (squash) May 30, 2023 22:47
@rodireich rodireich merged commit c1ea190 into master May 30, 2023
@rodireich rodireich deleted the 26509-query-and-log-results-for-last-full-vacuum branch May 30, 2023 22:48
marcosmarxm pushed a commit to natalia-miinto/airbyte that referenced this pull request Jun 8, 2023
* Log full vacuum info

* sanity

* commit change to force CI check

* improve queries

* Improve warning

* bump version and release note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Query and log results for last full vacuum
3 participants