Explore the possibilities for enriching column (and table) level metadata source systems hosted on AP

[LavMatt]

User Story

As a find moj data user
I want data from source systems (the curated AP databases in DE terminology) in the catalogue to be described
So that i can understand what data the are

Value / Purpose

Better descriptions of data brings more users

Useful Contacts

DMET colleagues

User Types

No response

Hypothesis

If we improve descriptions
Then we get more users and more feedback/engagement to improve content

Proposal

Work/speak with lead data/analytics engineers in DMET and gauge whether there are any metadata that could be ingested from source systems. Similar to metadata we have for nomis - it was mentioned there may be metadata for other source databases that we could use.

Could also gauge whether any data discovery tool data could be fed back into aws via DE pipelines to be available in Glue and via our Cadet ingestion of the generated models (which generate dbt models for source databases

Additional Information

There was some previous related work done and documented here #921

Chart shows the percentage of columns with descriptions in all tables of each source database we have in find moj data

Definition of Done

• Understand if there is any low hanging metadata that could be got to fill some metadata gaps as detailed above
• If yes to above plan out how the ingestions could be implemented

[LavMatt]

here's the table description equivalent chart