QEP 58: QGIS Styles, Symbols, and SVG Markers Sharing Repository

[akbargumbira]

QGIS Enhancement 58: QGIS Styles, Symbols, and SVG Markers Sharing Repository

Date 2016/04/27

Author Akbar Gumbira (@akbargumbira)

Contact akbargumbira@gmail.com

Status Draft

Version QGIS X.X

Summary

In this GSoC project, I will focus on making styles (.qml), symbols definition (.xml), and SVG markers sharing possible through web services that later can be accessed and used in QGIS. The project scope includes:

• Developing a web application to allow QGIS users (through OSGeo LDAP account) to upload their styles, symbols, and SVG’s.
• Integration in QGIS to allow users to fetch those resources and use them.

This project was partly done (styles and symbols sharing) in GSOC 2013, but unfortunately, the work was not merged in to master as some QGIS core developers thought that it was not production-ready, mainly on the server side. For the server side, I have moved the previous GSOC 2013 work to this repository: http://github.com/akbargumbira/QGIS-Sharing and will continue to work there.

GSoC 2013 Work

This is the work done on GSoC 2013. For symbol, user uploads xml file itself and the list of tags. For the styles, user submits a form with name, QML file, description, and tags.

• We can upload multiple symbols from one xml (we can export multiple symbols to one xml). In this case, it will create more than one row to the table.
  

Proposed Solution

Server Side

Authentication and user permissions

The behavior for authentication will follow what we have in Django plugin. User logs in using OSGeo acount.

User Management

• Admin/staff/superuser can:
  • trust a user. Resources that are uploaded by trusted users will not need approval from the admin,
  • untrust a user,
  • block a user. User would not be able to log in to the system (set is_active to False),
  • unblock a user.

Resource Management

• All users can:
  • browse, view, rate, and download public resources,
  • upload resources (after log in),
• Author can:
  • edit and remove their own resources
• Admin/staff/superuser can:
  • Approve and disapprove resources

Resource Classification

• Resource category will be hierarchy
• A resource (style, symbol, SVG marker, etc.) belongs to one or more categories
• User should be able to download a whole branch of the category tree
• User should be able to download a whole set of items sharing the same tag

Model

This section explains about the proposed model on the server side.

Author

Field Name	Type	Description
user	OneToOneField	One to One relationship to user table (any active user model defined in the django project)
is_trusted	BooleanField	Whether the author is trusted or not. If it is true, then the author can upload and publish it directly. If not, admin will check the uploaded resources and approve it.

Notes:

• Author row will be added after user uploads their first resource (symbol/style/svg). So, for every user, the first time they upload a resource, they will not be trusted and their resources will need to be approved by admin.

Category

Category will be hierarchical using django-mptt module (https://github.com/django-mptt/django-mptt)

Field Name	Type	Description
name	CharField	The name of a category (e.g Transportation, Health, Military, etc)
parent	TreeForeignKey	The parent of the category

Notes:

• Resource can have many categories. Category hierarchy could only be managed by admin.

Resource

This is an abstract model.

Field Name	Type	Description
name	CharField	The name of the resource
author	ForeignKey to Author	The author of the resource
categories	TreeManyToManyField to Category	The categories of the resource
tags	TaggableManager	Tags specified by the author. Using django tag app
description	TextField	The description of the resource
rating	AnonymousRatingField	Using django rating app
created_on	DateTimeFIeld	Date and time the resource is uploaded
is_approved	BooleanField	Whether the resource is approved/not

Symbol

This model inherits Resource abstract model.

Field Name	Type	Description
symbol_name	CharField	Extracted from uploaded XML
xml	TextField	The xml content of a symbol node
type	CharField	The type of the symbol or colorramp. If it’s symbol, the values could be:Marker, Line, Fill. If colorramps, the values could be: Gradient
is_symbol	Property	Return true if it’s symbol. False if it’s colorrramp (derived from the type value)

Notes:

• To upload symbol(s), User submit a form with:
  • Name
  • An XML file (that could define multiple symbols)
  • Category (user choose defined category)
  • Tags (users type tags as they want)
  • Description
• One XML file could contain multiple symbols. All the extracted symbols from this XML (that defines multiple symbols) will have the same properties of name, category, tags, and description. Authors later could edit those properties if they want.

Styles

This model inherits Resource abstract model.

Field Name	Type	Description
qml	FileFIeld	The location to the qml file
qgis_version	CharField	Extracted from QML.
renderer_type	CharField	Extracted from QML’s rendered node’s type value.
min_scale	FloatField	Extracted from QML’s minimumScale value
max_scale	FloatField	Extracted from QML’s maximumScale value
min_label_scale	FloatField	Extracted from QML’s minLabelScale value
max_label_scale	FloatField	Extracted from QML’s maxLabelScale value
scale_flag	BooleanField	Extracted from QML’s hasScaleBasedVisibilityFlag
label_scale_flag	BooleanField	Extracted from QML’s scaleBasedLabelVisibilityFlag

Notes:

• To upload style, User submits a form with:
  • Name
  • QML style file
  • Description
  • Category (users choose defined category)
  • Tags (users type tags as they want)

SVG Markers

This model inherits Resource abstract model.

Field Name	Type	Description
svg	FileField	The location to the SVG
is_authorized	BooleanFIeld	To confirm that the uploader is the author of the SVG or has the rights to share the icon

Notes:

• To upload SVG, User submits a form with:
  • Name
  • SVG file
  • Description
  • Category (user chooses defined category)
  • Tags (user types tags as they want)
  • Checkbox is_authorized

Client Side - QGIS Desktop client (python plugin)

In QGIS Desktop, user should be able to:

• browse repository items (symbols, styles etc.) with preview (if possible)
• search/sort items by tag, category, name ...
• download and install selected items (from a search result)
• upload of styles and items

Notes from Mailing List

• ​Anita Graser (@anitagraser) : I think we need something similar to the plugin platform:
  • Anyone can upload but an admin has to check the uploads before they are made public.
  • Trusted users can upload and publish directly.
  • Users can ​vote on resources. (Resources being symbols, styles, or SVGs.)
  • There are tags to classify/search for resources.
• Alessandro Pasotti (@elpaso): The Python plugin repository has proven to be a valuable mean for sharing code and offer new functionalities, symbology is a fundamental part of a GIS system and system for sharing symbols would enhance user experience and improve cooperation between individuals.
  • create production-ready re-usable Django app for sharing symbols
  • create a Python plugin for browsing, uploading and downloading symbols from the repository
  • test and deploy on qgis.org
• Andreas Neumann (@andreasneumann):
  • Check openclipart.org for SVG markers
  • SVG scrubbers: https://github.com/codedread/scour for the uploaded SVG files to remove unused/unnecessary resources. It helps to keep the SVG icon files clean and small.
  • Some minimum quality standards:
    • Contains the necessary SVG parameters that allows replacing of fill, stroke, stroke-width and opacity
    • keywords and titles
    • author contact
    • source (if copied from another source)
    • checkbox where the uploading person confirms that (s)he is either the
    • original author of the symbol or has the rights to share the icon
    • should we propose a standard license on the icons?
• Richard Duivenvoorde (@rduivenvoorde):
  • Categorizing icons based on their themes (e.g oil-business icons)

Further Considerations

• Check Scour for SVG scrubber
• Web Service: REST or RPC?
• Integration in QGIS: with C++ in Style Manager or with Python like Python Plugin Manager?

Backwards Compatibility

User will need to upgrade to the QGIS target version to be able to use this new feature.

Votes

(required)

[timlinux]

Nice first draft, thanks @akbargumbira . @rduivenvoorde here is the info you were after regarding what is being planned for the style repo. Please add your thoughts if you have any.

[elpaso]

Thanks @akbargumbira

Here are few random thoughts:

Classification

• categories should be a hierarchy
• an item (style, symbol etc.) belongs to one or more categories
• the user should be able to download a whole branch of the category tree
• the user should be able to download a whole set of items sharing the same tag

Items model

• item name should probably have a unique constraint, or at least, must be unique in its categories (for multiple symbols XML we should automatically assign incremental number suffixes to the item name or use a similar system)
• do we really want to support private items?

Further considerations

• webservices: I'd suggest to use the same XML-RPC lib we are using on the plugins website
• for the client side, I strongly advice you to implement as a python plugin instead of a C++ one

QGIS Desktop client (python plugin)

The specs for the client GUI side are still missing, we should try to sketch them out ASAP (to the top of my mind @anitagraser , @NathanW2, @nyalldawson and @rduivenvoorde could probably give more hints on this)
Here is a first incomplete list of functions I'd like to see in the plugin:

• browse repository items (symbols, styles etc.) with preview (if possible)
• search/sort items by tag, category, name ...
• download and install selected items (from a search result)
• upload of styles and items

Authentication and user permissions

I believe that you can copy/paste the models, permissions config and authentication stuff from the plugins repo, we are using osgeo id on the plugins website, I'd like to see that kind of authentication here too.

Caching

Please note that this repo will (hopefully) have a heavy traffic, caching should be activated wherever it makes sense. Avoid query string params and use path parameters when possible.

Backwards Compatibility

I'd set a target for current master or 2.16 but if we are going to implement the client as a python plugin there shouldn't be any problem in supporting 2.14 too.

Validation

In the plugin website, we've been using a validator class to check for minimum requirements on the submitted plugins. I'd suggest to do the same here and to make this an independent package, so that we can include it in the client plugin. This will make much more easier for a style/symbol submitter to check if their work is suitable for upload on our repo before they actually submit them.

[pcav]

My thoughts:

• free tags have the potential of creating lots of noise; perhaps better to first suggest a lit of existing or default ones, not encouraging too much the addition of new ones
• there should be a method to mass import entire sets from other sources
• a licence and author should be made explicit
• is there a chance of adding malicious code to svg/xml? If so, a filter/check should be added.
  Thanks!

[elpaso]

@pcav yes, a list of available tags is a must have
Can you explain what you mean with "mass import entire sets...": from what kind of sources?
And yes, a validator routine with sanity checks is definitely required.

[pcav]

@elpaso there are nice symbol sets around with free licence (I have opened a ticket listing some of the potentially useful ones, I can search it back when useful). It would be good to have the possibility of importing the full set, maybe as a zipfile.

[NathanW2]

I need to sit down and read this fully yet but just on this note:

QGIS Desktop client (python plugin)
No plugins please Python or C++. What ever we come up with needs to be
baked into QGIS as a fire class core feature. The style manager could be
extended to provide download ability or we build a whole new tool for it,
but no plugins.

On Thu, Apr 28, 2016 at 8:30 PM, Paolo Cavallini notifications@github.com
wrote:

@elpaso https://github.com/elpaso there are nice symbol sets around
with free licence (I have opened a ticket listing some of the potentially
useful ones, I can search it back when useful). It would be good to have
the possibility of importing the full set, maybe as a zipfile.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#58 (comment)

[NathanW2]

first class**

On Thu, Apr 28, 2016 at 8:35 PM, Nathan Woodrow madmanwoo@gmail.com wrote:

I need to sit down and read this fully yet but just on this note:

QGIS Desktop client (python plugin)
No plugins please Python or C++. What ever we come up with needs to be
baked into QGIS as a fire class core feature. The style manager could be
extended to provide download ability or we build a whole new tool for it,
but no plugins.

On Thu, Apr 28, 2016 at 8:30 PM, Paolo Cavallini <notifications@github.com

wrote:

@elpaso https://github.com/elpaso there are nice symbol sets around
with free licence (I have opened a ticket listing some of the potentially
useful ones, I can search it back when useful). It would be good to have
the possibility of importing the full set, maybe as a zipfile.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#58 (comment)

[alexbruy]

Maybe better not to limit this proposal to styles and symbols, but also leave room for other resources like scripts, models, composer templates, etc?

[NathanW2]

Yeah, maybe something like a generic "Online Resources" thing.

On Thu, Apr 28, 2016 at 8:58 PM, Alexander Bruy notifications@github.com
wrote:

Maybe better not to limit this proposal to styles and symbols, but also
leave room for other resources like scripts, models, composer templates,
etc?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#58 (comment)

[elpaso]

@NathanW2 what's wrong with a Python plugin implementation? I was thinking at something like the plugin manager, a core plugin written in Python.
But if @akbargumbira thinks has the time and skills to do that in C++ I'm not against it.

@alexbruy and all: please note that this is GSOC project, with limited time and resources, IMHO we have better chances to have a useful outcome if we keep the scope limited.

[NathanW2]

Ok if it's a skill/time thing. I will review this QEP and get back back with any notes. What do you think should be the main focus on this GSOC. Is there more benefit on the server side and leaving the client. The client should be easy-ish provided the API from the server is easy to use.

[elpaso]

I'm pretty confident that @akbargumbira , with the help and support of the QGIS community will have enough time to build production-ready symbols/styles server and client and to test-deploy it.
The server side is mostly defined, the client is not, we have time until the 23 may to define the specs for the client and your help and suggestions are highly appreciated.
I feel that implementing the client side in Python would be much more easier and quicker, that's why I'm pushing in that direction.
For the server services, If the time is not enough, we could always add more items (scripts etc., like suggested by @alexbruy) later.

[alexbruy]

@elpaso my comment was not about implementing support for all kinds of resources during GSoC, but mainly about developing extensible platform, so support of other resources can be added easily later.

[akbargumbira]

@elpaso:

categories should be a hierarchy
an item (style, symbol etc.) belongs to one or more categories

Noted, I'll probably just use django-mptt (https://github.com/django-mptt/django-mptt) for that

item name should probably have a unique constraint, or at least, must be unique in its categories (for multiple symbols XML we should automatically assign incremental number suffixes to the item name or use a similar system)

I am not really sure about this that the name should be unique. i think resources are likely to have similar/the same name, e.g for SVG, we could expect that there will be more than one SVG for hospital, right?

do we really want to support private items?

Now that I think about the purpose of this repository, I think it's not necessary :) I'll remove it from the model

The specs for the client GUI side are still missing, we should try to sketch them out ASAP

Yes, I'll make the mock-up this weekend following the feedback given from this QEP

I believe that you can copy/paste the models, permissions config and authentication stuff from the plugins repo, we are using osgeo id on the plugins website, I'd like to see that kind of authentication here too.

I am going to stick to the model above instead of using the scheme from plugins repo. I would like to have is_trusted as a field instead of permission (For the resources, I don't want to have 3 different permissions for each type of resources. When someone gets trusted, it means (s)he could upload it directly all type of resources. I don't like global permission either, hence I added that field). But apart from that, I think more or less I could copy from plugin project for authentication. I'll start to put some layout code including this, this weekend.

Please note that this repo will (hopefully) have a heavy traffic, caching should be activated wherever it makes sense. Avoid query string params and use path parameters when possible.

Ok. Noted

In the plugin website, we've been using a validator class to check for minimum requirements on the submitted plugins. I'd suggest to do the same here and to make this an independent package, so that we can include it in the client plugin. This will make much more easier for a style/symbol submitter to check if their work is suitable for upload on our repo before they actually submit them.

Do you suggest to add a field like 'minimum_qgis_version' for the Resource model above and user needs to specify this when uploading the resources? For style, we can extract 'qgis_version' from the qml file though it only represents the QGIS version used when importing the style

@pcav

My thoughts:

free tags have the potential of creating lots of noise; perhaps better to first suggest a lit of existing or default ones, not encouraging too much the addition of new ones

One resource has categories (which will be chosen from tree structure) and free tags. Do you suggest to remove free tags from the model?

there should be a method to mass import entire sets from other sources

Noted. But I won't make this as a high priority for GSoC duration. I'm not going anywhere and probably could continue the work after GSoC on my free time.

a licence and author should be made explicit

For license, do you suggest to add it for SVG or also for style (.qml) and symbol definition (.xml) as well (these both are imported from QGIS)?

is there a chance of adding malicious code to svg/xml? If so, a filter/check should be added. Thanks!

Noted. I'll explore more about this first.

@NathanW2 what's wrong with a Python plugin implementation? I was thinking at something like the plugin manager, a core plugin written in Python.
But if @akbargumbira thinks has the time and skills to do that in C++ I'm not against it.

Personally I prefer more developing with Python (I am not fluent in C++ and it's been a long time I worked with it).

@alexbruy yes, I'll keep in mind that this platform should be able to be extended with other kind of resources.

[nyalldawson]

No plugins please Python or C++. What ever we come up with needs to be
baked into QGIS as a fire class core feature.

I'm with Nathan here - I'm ok if this is done in python, but please DON'T make it a plugin. It should be part of core just like the python console.

[m-kuhn]

free tags have the potential of creating lots of noise; perhaps better to first suggest a lit of existing or default ones, not encouraging too much the addition of new ones

One resource has categories (which will be chosen from tree structure) and free tags. Do you suggest to remove free tags from the model?

I think the way stackexchange does it is great. It's free form but proposes tags and offers additional descriptions for tags (tooltips). I would rather remove categories.

[m-kuhn]

Very nice to read this draft, it will be great to see this improvement.

I'm with Nathan here - I'm ok if this is done in python, but please DON'T make it a plugin. It should be part of core just like the python console.

I agree, this should definitely be core-functionality which is not opt-in like plugins.

If this is mostly implemented in python, it would be nice if the gui libraries don't depend on python code and the setup code is run in app. (I.e. it should be done the console way and not the edit forms way)

It would be great if Python 3 and Qt 5 could be taken into consideration right from the beginning. There are a couple of technologies which are deprecated and could potentially be used (e.g. don't use QtScript) just to be sure we don't implement things now on technology that is soon going to disappear.

On the other hand others require new Qt versions that we don't yet support but that could be interesting (e.g. http://doc.qt.io/qt-5/json.html). If there is use for it, I think it can be discussed to do so. Basically depends on how you feel it @akbargumbira =)

Looking forward to downloading a lot of beautiful symbols from there!

[elpaso]

@m-kuhn I also think that this must be in the core and not an optional feature. But I'd prefer a python implementation. What I had in mind was a core python plugin, just like "plugin manager", but if there are any other ways to do a python implementation that is not a (core) plugin it's fine for me.

@akbargumbira for the permissions: you can have 3 (or more) different permissions and set/unset them as a whole, but it's definitely up to you, just note that if you choose the user model way you'll need a custom user model, while if you stick with the Django permission model you can just use its API.

For the validator, let me try to explain what I mean:
we do need validation for the uploaded items, there will probably be a set of rules that the uploaded asset should pass before we decided if it is valid and accept it in the repo. If you design this validation routine as an independent module, you can re-use it in the client (the QGIS side) to perform the checks before the user tries to upload the asset to the repo. This would be very useful for the users that will know in advance if their symbol/style will be accepted in the repo and they will able to fix any error/problem in advance.

[NathanW2]

I think it was just a mix up of terms. I'm fine with a Python version e.g
plugin manager, console styles, they are just features that are called from
core code. Unlike a plugin that is handled generic for all and can be
turned on and off.

So +1 to do it in Python for it.

On Fri, 29 Apr 2016 10:38 pm Alessandro Pasotti notifications@github.com
wrote:

@m-kuhn https://github.com/m-kuhn I also think that this must be in the
core and not an optional feature. But I'd prefer a python
implementation. What I had in mind was a core python plugin, just like
"plugin manager", but if there are any other ways to do a python
implementation that is not a (core) plugin it's fine for me.

@akbargumbira https://github.com/akbargumbira for the permissions: you
can have 3 (or more) different permissions and set/unset them as a whole,
but it's definitely up to you, just note that if you choose the user model
way you'll need a custom user model, while if you stick with the Django
permission model you can just use its API.

For the validator, let me try to explain what I mean:
we do need validation for the uploaded items, there will probably be a set
of rules that the uploaded asset should pass before we decided if it is
valid and accept it in the repo. If you design this validation routine as
an independent module, you can re-use it in the client (the QGIS side) to
perform the checks before the user tries to upload the asset to the
repo. This would be very useful for the users that will know in advance if
their symbol/style will be accepted in the repo and they will able to fix
any error/problem in advance.

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#58 (comment)

[elpaso]

@NathanW2 fully agreed, sorry for being not clear with the terms.

[rduivenvoorde]

Some wild brainstorming here

I'm probably (too) late, but I'm still not sure which main goal we are actually have here:

• trying to create a nice icon-collection, and make it easily available to (Q)GIS users?
• create a webservice with a lot of metadata so it is easier to search for certain icons sets?
• making it possible for QGIS to easily load icon styles?
• what is the relation between the qml-styles and the actual icons?
• I think the current plans (for the 'serverside') are pretty complex aka need a lot of logic/knowledge/support
  (OR/off course all of this)

What about keeping the client (QGIS) side then as simple as possible (so no actual extra gui stuff there?), for example: for QGIS you just download a zip with icons, unzip them in a dir with the zipname, and put that dir in your svg-icon path. Nothing more: no reading of metadata. For QGIS the icons just stay 'dumb' image files (like the icons we currently have).

In that way we can cut up complexity for now, leaving (search, metadata, svg-checking...) complexity as much as possible on the server. In that way we also keep room for very (personal) simple icon-repo's (like github??) which for example only offer a couple of thematic zip's.

There are a lot of use-cases with: icon-theme's, searchable icon libraries, dynamic icon building repo's (in which you can choose certain styling/coloring scheme's) etc. We should not try to bring all this knowledge into QGIS in my opinion... Will the average user for example care about the license for an individual icon? In my experience he/she is just looking/searching for a zip with a coherent set of icons usable for his/her current GIS-work

/end brainstorm, feel free to ignore

[anitagraser]

In my opinion, the main goal of this work is to create an easy way for users to share and access symbols (SVGs) and styles (QML). Currently, it is only possible to load a symbol XML from URL/file path using Style Manager:

The issue here (besides lack of SVG and QML support) is discoverability. Users have to know that a specific symbol XML exists and where to get it. A new tool should therefore make it easy to discover available resources, e.g. suggest most popular resources and support search by name, tags, author, ...

Imho, the GUI should definitely display previews.

(Sidenote: There is already a solution to share Processing Python scripts in the Processing toolbox.)

[timlinux]

One other thing to think about is the possibility of project portability. I'm thinking of the use case here where a user creates a project with a bunch of downloaded symbols and then uploads it to the server. It would be very nice if any published symbols were just automatically fetched if they are not present on the file system when the project is loaded. We could have a GUID for each symbol so that it could be retrieved canonically.

Amongst @rduivenvoorde's brain storming ideas, I find it quite intriguing the idea of having a distributed repo like system so that users can host their own public symbology repos. Though I guess the complexity goes up a whole heap if we do something like that so the work may not be worth the reward...unless some how @akbargumbira can come up way to make such a thing really simple and clear.

[DelazJ]

feature implemented so maybe, should we close this QEP

[nyalldawson]

This QEP is being archived in order to empty the issue tracker on this repository. Previous discussion and voting on the QEP remains valid and unchanged.