Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding advanced database logging for PgSnapshot module #67

Merged
merged 2 commits into from
Jun 2, 2020

Conversation

mgcuthbert
Copy link
Contributor

This PR will add two types of logging into the PgSnapshot module in Osmosis.

The first type of logging is always performed and will create a summary of all the entities that have been updated in the database. It will include the following fields:

  • nodes_added
  • nodes_modified
  • nodes_deleted
  • ways_added
  • ways_modified
  • ways_deleted
  • relations_added
  • relations_modified
  • relations_deleted
  • changesets_applied
  • earliest_timestamp
  • latest_timestamp

This will give you a basic overview of every single update in the database and help debug issues if there are any inconsistencies in the database.

The second big of logging is a bit more advanced and will create a lot of noise/data in the database so only should be used be debugging a specific issue. This will capture all sql queries executed on the database and store in the sql_changes table. An alternate way to do this would be to capture all the sql queries that hit the Postgres database.

Otherwise you would use something like this:
./osmosis --read-xml-change <INPUT> --write-pgsql-change <INPUT> logging=true

@migurski
Copy link
Collaborator

migurski commented Jun 2, 2020

Nice, merging!

Since logging=true is a new command line switch, this will trigger a minor version update to 0.49.

@migurski migurski merged commit fb5f1e5 into openstreetmap:master Jun 2, 2020
@migurski
Copy link
Collaborator

migurski commented Jun 2, 2020

@mgcuthbert, would you be able to help with cloning and updating the wiki docs for a new major version number? Brett’s also thinking about moving primary documentation into the repo here, which may make this easier in the future.

@mmd-osm
Copy link

mmd-osm commented Sep 26, 2020

It seems this feature is causing some regressions for some users out there: https://help.openstreetmap.org/questions/76837/osmosis-error-relation-replication_changes-does-not-exist

@mgcuthbert mgcuthbert mentioned this pull request Sep 28, 2020
@mgcuthbert
Copy link
Contributor Author

Thanks, please see this PR #76 for addressing that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants