Update Datahub to 0.15

[MatMoore]

This needs testing on dev first.

Releases - https://github.com/acryldata/datahub/releases

Looks as though the PR has been included in RC6 - acryldata/datahub@v1.0.0rc5...v1.0.0rc6

Upgrade Docs

• https://datahubproject.io/docs/how/updating-datahub/#0150

Currently we're on 0.14.1, but now 0.15 is available.

This includes a new DataHubGc source which can clean up unwanted DataProcessInstance records.

In our case

• DataProcessInstance metadata before 13th Jan is unreliable for dbt, due to us ingesting run results from the "deploy docs" job. So it may be worth setting dataprocess_cleanup.retention_days temporarily to clean this up.
• We should configure a keep_last_n settings so that we don't run out of disk space

[mitchdawson1982]

Current status -

• Dev is running the latest version V0.15.0.1
• Ran ingestion with the updated cli version and the cadet based ingestions currently fail.

[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for workforce_stats
[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for workforce_stats
[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for bold_sm_spells
[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for sentences
[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for sentences
[2025-02-17 22:02:08,456] WARNING {root:243} - No top level tags found in database metadata file for derived_oasys_data_first
[2025-02-17 22:02:08,457] WARNING {root:243} - No top level tags found in database metadata file for derived_oasys_data_first
[2025-02-17 22:02:08,765] INFO {datahub.ingestion.run.pipeline:574} - Processing commit request for DatahubIngestionCheckpointingProvider. Commit policy = CommitPolicy.ALWAYS, has_errors=False, has_warnings=False
[2025-02-17 22:02:08,765] WARNING {datahub.ingestion.source.state_provider.datahub_ingestion_checkpointing_provider:99} - No state available to commit for DatahubIngestionCheckpointingProvider
[2025-02-17 22:02:08,770] INFO {datahub.ingestion.run.pipeline:594} - Successfully committed changes for DatahubIngestionCheckpointingProvider.
[2025-02-17 22:02:47,734] ERROR {datahub.ingestion.run.pipeline:78} - failed to write record with workunit urn:li:container:4d0d7fb5585c0a94e535cce7725324dc-containerProperties with ('Unable to emit metadata to DataHub GMS: com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries', 'status': 500, 'urn': 'urn:li:container:4d0d7fb5585c0a94e535cce7725324dc', 'workunit_id': 'urn:li:container:4d0d7fb5585c0a94e535cce7725324dc-containerProperties'}) and info {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries', 'status': 500, 'urn': 'urn:li:container:4d0d7fb5585c0a94e535cce7725324dc', 'workunit_id': 'urn:li:container:4d0d7fb5585c0a94e535cce7725324dc-containerProperties'}

Possible issue - datahub-project/datahub#12202

[mitchdawson1982]

PR for Issue has been merged, no release as yet.