Advanced export functionalities

[MBueschelberger]

Changes from this PR:

Previously, the parsed metadata from the source file was represented as a flat dictionary, which will be used e.g. by the dsms python sdk in order to push the entries of the custom properties field.

E.g.

assert pipeline.plain_metadata == {
 'SampleIdentifier-2': 'Probentyp_2',
 'OriginalGaugeLength': 80,
}

This plain_metadata was previously dynamically flattened from the pipeline.general_metadata:

assert pipeline.general_metadata == [
PropertyGraph(
	iri=https://w3id.org/steel/ProcessOntology/SampleIdentifier-2,
	suffix=SampleIdentifier-2,
	key=Probenkennung 2,
	value=Probentyp_2,
	annotation=None,
	value_relation=rdfs:label),
 QuantityGraph(
	iri=https://w3id.org/steel/ProcessOntology/OriginalGaugeLength,
	suffix=OriginalGaugeLength,
	key=Messlänge Standardweg,
	unit=http://qudt.org/vocab/unit/MilliM,
	value=80,
	unit_relation=qudt:hasUnit,
	value_relation=qudt:value),
]

With this release, the plain_metadata is depricated and replaced by to_dict, which can use an additional callable as kwarg in order to transform the schema into a desired shape:

assert pipline.to_dict() == {
"SampleIdentifier-2": {
              "label": "SampleIdentifier-2",
              "value": "Probentyp_2",
          },
"OriginalGaugeLength":  {
            "label": "OriginalGaugeLength",
            "measurement_unit": {
                "iri": "http://qudt.org/vocab/unit/MilliM",
                "label": "Millimetre",
                "namespace": "http://qudt.org/vocab/unit",
                "symbol": "mm",
            },
            "value": 80,
        },
 }
 

with schema-kwarg:

from dsms.knowledge.utils import sectionize_metadata

pipeline.to_dict(schema=sectionize_metadata) == {
    "sections": [
        {
            "entries": [
                {
                   "id": "id1733827725155y6c07h"
                   "label": "SampleIdentifier-2",
                    "value": "Probentyp_2",
                },
                {
                    "id": "id1733827725155ob0izy"
                    "label": "OriginalGaugeLength",
                    "measurement_unit": {
                        "iri": "http://qudt.org/vocab/unit/MilliM",
                        "label": "Millimetre",
                        "namespace": "http://qudt.org/vocab/unit",
                        "symbol": "mm",
                    },
            ],
            "name": "General",
            "id":  "id1733827725155yhxu2o"
        },
    ],
}

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
data2rdf
__init__.py	5	0	100%
config.py	27	1	1	96%
utils.py	12	2	2	83%
warnings.py	3	0	100%
data2rdf/models
__init__.py	3	0	100%
base.py	53	5	5	91%
graph.py	144	8	8	94%
mapping.py	47	1	1	98%
utils.py	58	14	14	76%
data2rdf/modes
__init__.py	4	0	100%
data2rdf/parsers
__init__.py	6	0	100%
base.py	153	17	17	89%
csv.py	168	20	20	88%
excel.py	183	17	17	91%
json.py	229	39	39	83%
utils.py	90	10	10	89%
data2rdf/pipelines
__init__.py	2	0	100%
main.py	95	16	16	83%
data2rdf/qudt
__init__.py	0	0	100%
utils.py	55	16	16	71%
TOTAL	1337	166	88%

Tests	Skipped	Failures	Errors	Time
121	0 💤	0 ❌	0 🔥	3m 28s ⏱️

[yoavnash]

This is a comment for the SDK, but I’d like to mention it here as well: I think a better name for sectionize_metadata would be CustomPropertiesSchema (I imagine camel case fits better here than snake case).

[MBueschelberger]

This is a comment for the SDK, but I’d like to mention it here as well: I think a better name for sectionize_metadata would be CustomPropertiesSchema (I imagine camel case fits better here than snake case).

why camel case? It is a method and hence the convention is snake case.

[yoavnash]

This is a comment for the SDK, but I’d like to mention it here as well: I think a better name for sectionize_metadata would be CustomPropertiesSchema (I imagine camel case fits better here than snake case).

why camel case? It is a method and hence the convention is snake case.

I'm not sure why I didn't use snake case, but the point still holds.

[MBueschelberger]

This is a comment for the SDK, but I’d like to mention it here as well: I think a better name for sectionize_metadata would be CustomPropertiesSchema (I imagine camel case fits better here than snake case).

why camel case? It is a method and hence the convention is snake case.

I'm not sure why I didn't use snake case, but the point still holds.

sure, the name makes sense and I would rename it in the SDK. Just was wondering about the case schema.

[MBueschelberger]

This is a comment for the SDK, but I’d like to mention it here as well: I think a better name for sectionize_metadata would be CustomPropertiesSchema (I imagine camel case fits better here than snake case).

why camel case? It is a method and hence the convention is snake case.

I'm not sure why I didn't use snake case, but the point still holds.

sure, the name makes sense and I would rename it in the SDK. Just was wondering about the case schema.

Maybe make_custom_properties_schema would be even better, because it's a function?