FOGL-6719 - Deprecate asset record handling in Rest API

[ashish-jabble]

Items

• Add new column deprecated_ts in asset_tracker table; DB schema addition for SQLite
• Add new column deprecated_ts in asset_tracker table; DB schema addition for SQLitelb - Functional testing BLOCKED BY FOGL-6742
• Add new column deprecated_ts in asset_tracker table; DB schema addition for PostgreSQL - Functional testing BLOCKED BY FOGL-6742
• Add new API for asset_tracker to update the deprecated_ts via PUT only as we are not deleting anything from the table. You can call curl -sX PUT http://localhost:8081/fledge/track/service/XXX/asset/XXX/event/XXXX
• GET track entry has now with deprecated_ts KV pair too
• South API changes - Seems like there is no support available in Storage layer for is NULL, Therefore pending task. SEE FOGL-6740 and FOGL-6741
• Filters API changes
• Service API changes
• Task API changes
• Asset browser API changes? - As per design, NO NEED TO TAKE ACTION; as record is expected to show on Asset and Readings page even if deprecated_ts is not null
• Unit test Fixes

[MarkRiddoch]

Looks good to me - thanks for getting this down quickly.

[mshariq-nerd]

@ashish-jabble What is the need of select query before update? we should see affected rows from storage result; otherwise this will slow down.

Also I observed, we can deprecate an already deprecated asset again. I think once deprecated, it should not change any more.
e.g.

 curl -sX PUT http://localhost:8081/fledge/track/service/sine/asset/sinusoid/event/Ingest | jq
{
  "success": "Asset record entry has been deprecated."
}

Now if we will run the PUT request again, it will execute successfully and update the deprecatedTimestamp.

[ashish-jabble]

@mshariq-nerd

Also I observed, we can deprecate an already deprecated asset again. I think once deprecated, it should not change any more.

This cannot be appear once FOGL-6740 is done. As there is no IS NULL support in storage layer. Once available I will update my query.

[mshariq-nerd]

@ashish-jabble seeing one issue

1. Add a south service e.g. sinusoid with asset name sinusoid (data Ingest started successfully)
2. Delete the service
3. Add the service again with same name and asset. e.g. sinusoid with asset name sinusoid
   Issue: Asset no longer available in the service

"services": [
{
"name": "sinusoid",
"address": "localhost",
"management_port": 35095,
"service_port": 36753,
"protocol": "[http](http://192.168.1.112:8081/fledge/http)",
"status": "running",
"assets": [],
"plugin": {
"name": "sinusoid",
"version": "1.9.2"
},
"schedule_enabled": true
}
]

[ashish-jabble]

@mshariq-nerd - I am not sure if that's an issue or its by design with the current implementation of deprecated asset record entry. So @MarkRiddoch can better confirm this.

AFAIK, Once an asset_track entry is deprecated (given the combination of plugin+ service + event + asset as by this we treat it as a unique record for an asset record) you will be no longer available to see the entry in south service API if that combination of asset tracker record appears again in the system. As we have changed the south API query by excluding the assets if deprecated_ts isnull or empty And this is as per JIRA requirements. See

In south.py change the query in _get_tracked_plugin_assets_and_readings to exclude any assets with a deprecated timestamp, i.e. update the query

[mshariq-nerd]

@ashish-jabble Not sure but I am not deprecating anything here. steps are simple add service, delete service and again add service with same name and asset. Now asset no longer appearing in service.

[ashish-jabble]

Not sure but I am not deprecating anything here. steps are simple add service, delete service and again add service with same name and asset. Now asset no longer appearing in service.

@mshariq-nerd As per JIRA have implemented the updation of deprecated_ts internally while deleting a service, task and filter. Also we have used for the extra SELECT query inside it, so FOGL-6749 is created for workarounds.

i.e On REST API endpoint like deleting a service or task or filter - we have internally updated the deprecated_ts for an asset record if exists.

[mshariq-nerd]

There is a more simple repro:

1. Disable south service
2. Purge asset on asset & reading page
3. Deprecate asset on south service page
4. Refresh, all gone - good
5. Enable the service again

Issue: Now asset no longer appearing in service.
/cc @ashwinscale