Migrate to GeoTools 23.0 #4488

jusabatier · 2020-03-02T11:07:25Z

Modifications for migrate to GeoTools 23.0

Compile OK with mvn clean install
Start without error in JAVA11 / Tomcat9

Still have to test :

JAVA8 compatibility
Usage tests

Related to #3882

fxprunayre · 2020-03-02T11:10:50Z

Don't forget to test WFS feature indexing which was not working (XY order issue) in our last try of updating GT (see #3886)

jusabatier · 2020-03-02T14:14:13Z

Don't understand how I can test it ... :/

fxprunayre · 2020-03-02T14:22:21Z

Don't understand how I can test it ... :/

https://geonetwork-opensource.org/manuals/trunk/en/user-guide/analyzing/data.html?highlight=wfs
The easiest is probably to:

install Elasticsearch https://geonetwork-opensource.org/manuals/trunk/en/maintainer-guide/installing/installing-index.html#installing-index
harvest a GeoServer node eg. https://demo.geo-solutions.it/geoserver
add a layer to the map, go to filter menu and trigger indexing.

jusabatier · 2020-03-02T15:03:01Z

Trying to install ES with your tuto, but where is supposed to be the es/es-config directory ?

fxprunayre · 2020-03-02T15:04:09Z

Trying to install ES with your tuto, but where is supposed to be the es/es-config directory ?

https://github.com/geonetwork/core-geonetwork/tree/master/es#install-configure-and-start-elasticsearch

Fixed errors in the doc and updated version number.

jusabatier · 2020-03-02T15:44:11Z

I installed ES, started it, loaded the indices, updated the URL in WEB-INF/config.properties and restarted it.

But when I add a layer to the map and go to filter tab, it says something like "Error accessing index" and I have not button to index the data.

fxprunayre · 2020-03-02T16:17:12Z

But when I add a layer to the map and go to filter tab, it says something like "Error accessing index" and I have not button to index the data.

What does
http://localhost:8080/geonetwork/warninghealthcheck (if running from source, you need to build and start the app with mvn jetty:run -Pes to load ES
http://localhost:9200/gn-features
http://localhost:8080/geonetwork/index/features?_=_search
return ?

jusabatier · 2020-03-02T16:26:42Z

I doesn't test it on localhost, I deploy the war on a VM with tomcat9.

http://localhost:8080/geonetwork/warninghealthcheck

= > Error creating bean with name 'EsSearchManager': Unsatisfied dependency expressed through field 'client': No qualifying bean of type [org.fao.geonet.es.EsClient] found for dependency [org.fao.geonet.es.EsClient]

fxprunayre · 2020-03-02T16:29:30Z

= > Error creating bean with name 'EsSearchManager': Unsatisfied dependency expressed through field 'client': No qualifying bean of type [org.fao.geonet.es.EsClient] found for dependency [org.fao.geonet.es.EsClient]

So you need to build the app with es profile first.

jusabatier · 2020-03-02T16:41:00Z

With mvn clean install -dskipTests -Pes ?

jusabatier · 2020-03-02T16:54:12Z

Ok it seems to progress :

http://localhost:8080/geonetwork/warninghealthcheck :

[
    {
    "name": "DashboardAppHealthCheck",
    "status": "ERROR",
    "msg": "Dashboard application is not available currently. This component is only required if you use dashboards."
  },
    {
    "name": "DeadlockedThreadsHealthCheck",
    "status": "OK"
  },
    {
    "name": "Free file handles",
    "status": "OK"
  },
    {
    "name": "FreeConnectionsHealthCheck",
    "status": "OK"
  },
    {
    "name": "HarvestersHealthCheck",
    "status": "OK"
  },
    {
    "name": "IndexHealthCheck",
    "status": "OK",
    "msg": "0 records indexed in remote index currently."
  },
    {
    "name": "NoIndexErrorsHealthCheck",
    "status": "OK"
  }
]

Now in filter tab it says something like "Entities not indexed", but I still not have any button to index them.

fxprunayre · 2020-03-02T17:06:49Z

With a metadata produced from OGC harvester using https://demo.geo-solutions.it/geoserver ?

Once harvested add USA pop on the map and you should be able to have

Click on a feature in the table should zoom to correct location

jusabatier · 2020-03-03T07:56:14Z

Yes.

I tried to harvest both WMS and WMTS, it creates a metadata for WMS/WFS service.

Here is a part the log for WFS :

2020-03-03 08:50:06,504 INFO [TestWFS] - Starting harvesting of TestWFS
2020-03-03 08:50:06,528 INFO [TestWFS] - Started harvesting from node : TestWFS (OgcWxSHarvester)
2020-03-03 08:50:06,528 INFO [TestWFS] - Retrieving remote metadata information for : TestWFS
2020-03-03 08:50:07,217 INFO [TestWFS] - - Number of layers, featureTypes, Coverages or process found : 53
2020-03-03 08:50:07,217 INFO [TestWFS] - - Loading layer: spearfish with UUID 342bb803677d1a5392b59779b3744719deff09d6 (hash of capabilities URL + layer name).
2020-03-03 08:50:07,217 INFO [TestWFS] - - Searching for metadataUrl for layer spearfish
2020-03-03 08:50:07,218 INFO [TestWFS] - - No metadataUrl attribute found for layer 'spearfish'
2020-03-03 08:50:07,253 WARN [TestWFS] - - Failed to load metadata document for layer 'spearfish'. Error is: 'For input string: ""'

I checked create metadata for each layer and it look like layers can't load, but the error message is not very explicit...

I also have a new tab "Data Harvester" which lists jobs for WFS indexations, but it's empty.

fxprunayre · 2020-03-03T08:26:43Z

I tried to harvest both WMS and WMTS, it creates a metadata for WMS/WFS service.

Can you check your harvester config, the following works fine for me

jusabatier · 2020-03-03T08:31:51Z

I use exactly the same, but still have no layers loaded with this warn in logs :

WARN [wms] - - Failed to load metadata document for layer 'spearfish'. Error is: 'For input string: ""',

fxprunayre · 2020-03-03T08:36:14Z

I use exactly the same, but still have no layers loaded with this warn in logs :

Bizarre. Works also fine on https://vanilla.geocat.net/geonetwork/srv/fre/admin.console#/harvest

Can you choose a User in the config ? Error is not super explicit - sounds similar to when the group was mandatory (but it is supposed to be solved in master branch)

jusabatier · 2020-03-03T08:44:30Z

Selecting a user solve the problem, I managed to load all layers, and when add one and go to filter tab, I can index entities.

After index entities, I have new filters on the layer based on fields.

jusabatier · 2020-03-03T08:46:37Z

I think problem you mentioned is not solved, because indexed entities are located in South Pole :(

fxprunayre · 2020-03-03T08:56:59Z

Selecting a user solve the problem, I managed to load all layers

Weird (I'll try to have a look to the user issue) and good you made harvesting work.

I think problem you mentioned is not solved, because indexed entities are located in South Pole :(

We tried a couple of options suggested by @jodygarnett #3886 (comment) with no luck in last Bolsena sprint. Would be good to manage to find a solution to move forward on this.

jusabatier · 2020-03-04T12:44:47Z

I tried use of WFSDataStoreFactory.AXIS_ORDER in WFSHarvesterExchangeState like suggested by @jodygarnett and it seems to work well with topp:states layer.

When clear and reindexing features, they are at the right place on the map, and I have no more ES error log related to Y value overflow.

jusabatier · 2020-03-04T12:58:13Z

But not work with layers over Italy like geosolutions:regioni...

jusabatier · 2020-03-05T14:51:08Z

After many tests, I manage to force axis order for WGS84 :

if( defaultCRS.equals(epsg4326) && firstAxis.getDirection().absolute() == AxisDirection.NORTH ) {
    config.setAxisOrder(WFSDataStoreFactory.AXIS_ORDER_EAST_NORTH);
    config.initDataStore();
}

And it handle corectly topp:states from https://demo.geo-solutions.it/geoserver

But when trying with geosolutions:regioni which have an EPSG:3044 CRS, it doesn't work.
With some logs, it appears that the code : https://github.com/geonetwork/core-geonetwork/blob/master/workers/wfsfeature-harvester/src/main/java/org/fao/geonet/harvester/wfsfeatures/worker/EsWFSFeatureIndexer.java#L254

Logs of features retieved with getFeatures show coords like this :

CRS Id : [EPSG:4326]
-- Feature Coord :
x => 4754785.01965227
y => 625766.32391491

The CRS is updated, but the coordinates still in EPSG:3044, no reprojection done.

Same problem with others SRS like EPSG:26713 of sf:roads layer.

jusabatier · 2020-03-06T10:09:55Z

WIth this commit, it works with both topp:states and geosolutions:regioni from https://demo.geo-solutions.it/geoserver

What I understand searching infos around the web :

WGS84 is the EPSG:4326 and is officially defined with latitude/longitude axis order (y/x)
Some applications expect WGS84 with longitude/latitude axis order (x/y), ElasticSearch is one of them
An unofficial CRS equivalent to WGS84 with longitude/latitude exists : "urn:ogc:def:crs:OGC:1.3:CRS84"
GeoServer is anarchic, it return both axis order, depends of the GML version and srsName parameter.

The problem was that the feature reprojection defined in setCoordinateSystemReproject function didn't work, so features sent to ElasticSearch were not in WGS84 but in native CRS.

Moreover, ElasticSearch expects WGS84 with longitude/latitude axis order (x/y), so the best way to get it is to :

Use the defaultCRS of the layer (aeb4ab8#diff-a92a6d5e6221144d4232310c576d6fb4R115)
Set this defaultCRS as the CRS of the query's returned features (aeb4ab8#diff-27df765efe58f8f003cd9dddd8893450R250)
Reproject those features to "urn:ogc:def:crs:OGC:1.3:CRS84" (aeb4ab8#diff-27df765efe58f8f003cd9dddd8893450R260-R262)

For the GeoServer anarchy, I think as the official WGS84 axis order is latitude/longitude (y/x), we should handle only this axis order...

This currently work because we use GML3 and no srsName param.
We should have a problem with axis order in the future if we use GML32 as it return bad axis order for WGS84 and no srsName param.

jodygarnett · 2020-03-25T14:14:04Z

As part of this bug hunt I documented what geoserver does here

TLDR: Each WFS / WMS specification changes what the EPSG codes mean ... and GeoServer does its best to keep up!

WFS servers (MapServer, deegree, GeoServer, ArcGIS) manage to have every variation of axis order incorrect .. so GeoNetwork will need to provide flags to allow harvester to be configured to connect to each service (there will never be a solution to this only configuration).

jodygarnett · 2020-03-25T14:15:50Z

@jusabatier see above, configuration options are required.

The previous geotools client did its best to "guess" workarounds; in practice this meant there were always WFS servers we could not talk to. The new client provides additional connection parameters.

...harvester/src/main/java/org/fao/geonet/harvester/wfsfeatures/worker/EsWFSFeatureIndexer.java

jodygarnett · 2020-04-15T18:53:00Z

I really like this approach, reproject whatever contents is returned to CRS:84 because we cannot trust the different WFS implementations.

It also looks like all features are being requested, so we do not need to worry about different implementations getting axis order confused on the request.

What is needed for this work to go ahead?

jodygarnett · 2020-04-16T15:09:11Z

For the GeoServer anarchy, I think as the official WGS84 axis order is latitude/longitude (y/x), we should handle only this axis order...

Putting in a kind word for GeoServer here, the anarchy is the specifications, it is actually a real bother to follow what they are doing and implement them "correctly".

If you stay with a specific protocol version (example WFS 2.0 + GML 3.2) the result is consistent. When you mix and match the result is the madness you describe :)

jodygarnett · 2020-04-16T23:40:16Z

This is looking good to me, can we rebase this PR and put it up for review?

jodygarnett · 2020-04-16T23:41:55Z

pom.xml

@@ -1443,7 +1446,7 @@

    <!-- NOTE: When updating GeoTools, check which version
    of Postgres is used and update pg.version if needed. -->
-    <geotools.version>22.3</geotools.version>
+    <geotools.version>23.0</geotools.version>


We should build once more when GeoTools is officially anounced, we are currently having a build issue due to the new osgeo repo ... performing faster and allowing multi-threaded builds to trip on themselves. The Maven local repository is not thread safe apparently ...

josegar74 · 2020-04-17T11:24:42Z

@jusabatier please if you can fix the conflicts, I'll check to test next week and merge it, thanks.

jusabatier · 2020-04-17T12:44:47Z

I fixed the conflicts.

Compile fine when skipping tests.

But when doing them I have an error :

[ERROR] Failures:
[ERROR] BatchOpsMetadatReindexerTest.asyncMultiThread:127 expected:<4> but was:<3>
[ERROR] Errors:
[ERROR] ApiImportSpatialDirectoryEntriesTest.nominal:38->processLayersZip:102 » NoSuchMethod
[ERROR] ApiImportSpatialDirectoryEntriesTest.withReprojTo3857:67->processLayersZip:102 » NoSuchMethod
[ERROR] ApiImportSpatialDirectoryEntriesTest.withReprojTo4326_IESame:52->processLayersZip:102 » NoSuchMethod

jodygarnett · 2020-04-17T16:41:02Z

NoSuchMethod seems like we need more information?

Looking at withReprojTo3857:

        processLayersZip("layers_111_feature_EPSG_3857.xml",
                new RequestCustomizer() {
                    @Override
                    public void customizeRequest(MockMultipartHttpServletRequest request) {
                        request.setParameter("schema", "iso19139");
                        request.setParameter("process", "build-extent-subtemplate");
                        request.setParameter("descriptionAttribute", "desc");
                        request.setParameter("onlyBoundingBox", "false");
                        request.setParameter("geomProjectionTo", "EPSG:3857");
                    }
                });

A MockMultipartHttpServletRequest is being setup, and I guess a call is being made to it that was not anticipated...

Looking at processLayersZip:102 shows:

        SimpleMetadataProcessingReport report = (SimpleMetadataProcessingReport) invoker.invoke(request, response);
        assertEquals(200, response.getStatus());
        assertEquals(3, report.

This is an example of mock testing being very fragile :P

josegar74 · 2020-04-20T09:03:07Z

I get this test error:

[ERROR] Failures: 
[ERROR]   MetadataApiTest.getRelated:581 Status expected:<200> but was:<400>
[INFO] 
[ERROR] Tests run: 377, Failures: 1, Errors: 0, Skipped: 12

With master branch seem ok, to check further.

jodygarnett · 2020-04-29T21:06:29Z

@josegar74 as in this:

        mockMvc.perform(get("/srv/api/records/" + MAIN_UUID + "/related")
            .session(mockHttpSession)
            .accept(MediaType.APPLICATION_JSON))
            .andExpect(status().isOk())
            .andExpect(content().contentType(API_JSON_EXPECTED_ENCODING));

Do you have the surefire results stack trace anywhere?

jodygarnett · 2020-04-29T21:52:38Z

@josegar74 running this locally works for me:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21:41 min
[INFO] Finished at: 2020-04-29T14:41:31-07:00
[INFO] ------------------------------------------------------------------------

josegar74

The issue with the test seemed some local issue, in travis is passing fine.

jodygarnett · 2020-05-04T18:34:03Z

Thank you, happy to see this go ahead

fxprunayre

LGTM. Merged in branch 4.0.x too. Thanks.

fxprunayre · 2020-06-16T15:55:59Z

It looks like on some setup this introduced this error (happens on different services validation, related resource) - may not be related to this update - I don't really know. On docker with tomcat:8-jre8, one app is running fine. Another one is not.

{"message":"NestedServletException","code":"runtime_exception","description":"Handler dispatch failed; 
nested exception is java.lang.NoClassDefFoundError: 
Could not initialize class com.sun.xml.bind.v2.model.impl.Utils"}

and sounds related to org.glassfish.jaxb:jaxb-runtime:jar

[^[[1;34mINFO^[[m] |  +- org.geotools:gt-wms:jar:23.0:compile
[^[[1;34mINFO^[[m] |  |  +- org.geotools:gt-coverage:jar:23.0:compile
[^[[1;34mINFO^[[m] |  |  |  +- it.geosolutions.imageio-ext:imageio-ext-tiff:jar:1.3.2:compile
[^[[1;34mINFO^[[m] |  |  |  |  +- it.geosolutions.imageio-ext:imageio-ext-utilities:jar:1.3.2:compile
[^[[1;34mINFO^[[m] |  |  |  |  \- it.geosolutions.imageio-ext:imageio-ext-geocore:jar:1.3.2:compile
[^[[1;34mINFO^[[m] |  |  |  |     +- javax.xml.bind:jaxb-api:jar:2.2.11:compile
[^[[1;34mINFO^[[m] |  |  |  |     +- it.geosolutions.imageio-ext:imageio-ext-streams:jar:1.3.2:compile
[^[[1;34mINFO^[[m] |  |  |  |     +- org.glassfish.jaxb:jaxb-runtime:jar:2.4.0-b180830.0438:runtime

The class is also provided by Java rt.jar.

Anyone observing this? Any suggestions?

jodygarnett · 2020-06-17T00:24:03Z

Anyone observing this? Any suggestions?

It is a common problem ... when upgrading to Java 11 (on newer systems you have to enable jaxb and javax.annotations).

Can you double check what version of Java you are starting up please @fxprunayre?

fxprunayre · 2020-06-17T05:38:20Z

What is weird is that it is same WAR based on master deployed on docker with tomcat:8-jre8. So I don't expect that this kind of errors could happen on some and not on other - We have 4 apps deployed for test/prod and only one is having the issue.

jusabatier · 2020-06-17T06:47:22Z

Generaly, it's due to the fact that JAXB APIs are considered to be Java EE APIs and therefore are no longer contained on the default classpath in Java SE 9. In Java 11, they are completely removed from the JDK.

So if you run it on JAVA 8, it should work, but not in JAVA 9 or above.

Here are some solutions : https://stackoverflow.com/questions/43574426/how-to-resolve-java-lang-noclassdeffounderror-javax-xml-bind-jaxbexception-in-j

pvgenuchten · 2020-06-22T21:44:27Z

is/will this be backported to 3.10?
i noticed (also in #4072) this patch is required for QGIS metasearch to query geonetwork with a spatial filter

josegar74 · 2020-06-23T07:51:19Z

No plans to backport this work to 3.10

* Migrate to GeoTools 22.3 * Correct feature reprojection in WFS indexation * Upgrade to GeoTools 23.0 * Remove unusefull query and add final statement to close iterator

Migrate to GeoTools 22.3

908df9b

jusabatier force-pushed the patch-geotools-22.3 branch from 5d99d81 to 908df9b Compare March 4, 2020 13:25

Correct feature reprojection in WFS indexation

aeb4ab8

jusabatier mentioned this pull request Mar 11, 2020

[Header] - Special characters display incorrectly with OpenJDK 11 georchestra/georchestra#2965

Closed

fxprunayre mentioned this pull request Mar 25, 2020

Geotools v21.4 upgrade #3886

Closed

jodygarnett reviewed Apr 15, 2020

View reviewed changes

...harvester/src/main/java/org/fao/geonet/harvester/wfsfeatures/worker/EsWFSFeatureIndexer.java Show resolved Hide resolved

Upgrade to GeoTools 23.0

5b1aa92

Remove unusefull query and add final statement to close iterator

b5c6385

jusabatier changed the title ~~Migrate to GeoTools 22.3~~ Migrate to GeoTools 23.0 Apr 16, 2020

jodygarnett reviewed Apr 16, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into patch-geotools-22.3

5d1adfa

josegar74 approved these changes May 4, 2020

View reviewed changes

josegar74 merged commit a0782c9 into geonetwork:master May 4, 2020

ianwallen mentioned this pull request May 4, 2020

Error starting geonetwork based on master branch in Windows. #4661

Closed

fxprunayre reviewed May 5, 2020

View reviewed changes

jusabatier mentioned this pull request May 5, 2020

Geonetwork - Support for JAVA 11 georchestra/georchestra#3051

Closed

josegar74 mentioned this pull request May 8, 2020

Analysis to support Java 11+ #4668

Open

pvgenuchten mentioned this pull request Jan 8, 2021

MetaSearch: "Could not create crs: urn:ogc:def:crs:OGC:1.3:CRS84" error when a bounding box is specified qgis/QGIS#40778

Closed

Migrate to GeoTools 23.0 #4488

Migrate to GeoTools 23.0 #4488

Conversation

jusabatier commented Mar 2, 2020 • edited Loading

fxprunayre commented Mar 2, 2020

jusabatier commented Mar 2, 2020

fxprunayre commented Mar 2, 2020

jusabatier commented Mar 2, 2020

fxprunayre commented Mar 2, 2020 • edited Loading

jusabatier commented Mar 2, 2020

fxprunayre commented Mar 2, 2020

jusabatier commented Mar 2, 2020

fxprunayre commented Mar 2, 2020

jusabatier commented Mar 2, 2020

jusabatier commented Mar 2, 2020

fxprunayre commented Mar 2, 2020 • edited Loading

jusabatier commented Mar 3, 2020

fxprunayre commented Mar 3, 2020

jusabatier commented Mar 3, 2020

fxprunayre commented Mar 3, 2020

jusabatier commented Mar 3, 2020

jusabatier commented Mar 3, 2020

fxprunayre commented Mar 3, 2020

jusabatier commented Mar 4, 2020

jusabatier commented Mar 4, 2020

jusabatier commented Mar 5, 2020

jusabatier commented Mar 6, 2020

jodygarnett commented Mar 25, 2020

jodygarnett commented Mar 25, 2020

jodygarnett commented Apr 15, 2020

jodygarnett commented Apr 16, 2020

jodygarnett commented Apr 16, 2020

jodygarnett Apr 16, 2020

Choose a reason for hiding this comment

josegar74 commented Apr 17, 2020

jusabatier commented Apr 17, 2020

jodygarnett commented Apr 17, 2020

josegar74 commented Apr 20, 2020

jodygarnett commented Apr 29, 2020

jodygarnett commented Apr 29, 2020

josegar74 left a comment

Choose a reason for hiding this comment

jodygarnett commented May 4, 2020

fxprunayre left a comment

Choose a reason for hiding this comment

fxprunayre commented Jun 16, 2020 • edited Loading

jodygarnett commented Jun 17, 2020

fxprunayre commented Jun 17, 2020

jusabatier commented Jun 17, 2020

pvgenuchten commented Jun 22, 2020

josegar74 commented Jun 23, 2020

jusabatier commented Mar 2, 2020 •

edited

Loading

fxprunayre commented Mar 2, 2020 •

edited

Loading

fxprunayre commented Mar 2, 2020 •

edited

Loading

fxprunayre commented Jun 16, 2020 •

edited

Loading