Finalise adding Entities to FMTM workflows & other tasks

[spwoodcock]

Is your feature request related to a problem? Please describe.

• Entities have been added to the project creation.
• We have implemented their use on the project details map too.
• There is still work to do to ensure Entities are fully integrated into FMTM and working for the 2024.03 release.

Describe the solution you'd like

• all remaining issues in https://github.com/hotosm/fmtm/milestone/28
• Loading of Entities via Intent from the frontend buttons should be tested thoroughly.
  • The custom APK will be required from here for testing https://github.com/hotosm/odkcollect/releases/tag/entity-preselection
• The mapper flow is now significantly improved & should be tested:
  • Feature mapping status shown via coloured buildings.
  • Geolocation and navigation to buildings working nicely.
  • Downloading of PMTiles archive for offline basemaps, see Creating PMTiles basemap requires creating MBTiles first #1466. MBTiles generation may have to be triggered before PMTiles, until the issue in osm-fieldwork basemapper PMTile creation is debugged and fixed.
  • A 'Go Offline' frontend button that prepares the users device for going offline, see Add 'Go Offline' functionality to project details page #1437.
• Using Entities will likely break usage of production projects: we need to decide if this is acceptable. There are a few strategies I can think of #1522
  1. Create migrations to generate Entities for the old projects (difficult), so they work as normal.
  2. Create a /projects/{project_id}/export endpoint and a /project/import endpoint for exporting/importing data (difficult).
  3. Export the submissions for production (beta) projects, save somewhere, then archive/delete the projects. Inform users that their submission data from the public beta is available if they contact us (easier).
• Project stats dashboard should use Entity data where appropriate, fallback to submission endpoints #1521
  • We still need to get the submissions from ODK, but only for download.
  • We probably no longer need to cache in the S3 either, as we use a single form for submissions.
  • Performance should be tested to check if we can just download from ODK and pass onto the user, within a reasonable timing of say <10s.
  • The frontend infographics / stats page should work with the refactored endpoints
• Time could be taken to refactor and remove redundant code from the frontend, to make it easier to navigate for new contributors.
• fmtm-splitter needs an update to fix the various issues.
• osm-fieldwork currently only has one Entities form for buildings. The archived XLSForms for other category types should be updated to also work with Entities. healthcare would be a good one to start on.

[Sujanadh]

I updated the endpoint to fetch the submissions correctly from central for now they are taking around 8 seconds to load submissions. And above, do you mean to call the entities directly from frontend? without involving the backend?

[spwoodcock]

The statistics page can pretty much just use the entities endpoints to provide the stats, like how many features mapped, the user that mapped them, the validation status etc.

All that info is attached to the Entity.

The submissions endpoints only really need to be used for the user to download the submission data as a file format like GeoJSON/CSV/etc.
It's acceptable to wait a bit for this download to complete.

By using entities on the submission / stats page we can significantly reduce loading time and no longer require caching of the submission. Does that make sense?

[Sujanadh]

But we also need submission endpoints for the bar graph and table, in addition to downloading them. right?
How do we count submissions from entities? by counting statuses?
But what if I change the entity status manually from the endpoint? (though rare case in a practical scenario.)

How do we get contributors from entities?

[spwoodcock]

Yep we can count the statuses = MAPPED or INVALID/BAD as the total number of submissions made.

I added further info the description of #1521 to clarify (the original task was badly worded)

[spwoodcock]

Closing as most tasks are complete & QA Ready. The remainder has been moved to 2024.04 milestone tasks.