What's the proper way to install GeoNode core for development?

[gbassiere]

I'm running Ubuntu and installed all necessary packages, I have both Postgresql 14 and Tomcat 9 up and running on localhost. Geoserver works fine. Then:

    mkvirtualenv geonode
    workon geonode
    git clone https://github.com/GeoNode/geonode.git
    pip install -e geonode
    cd geonode/geonode
    cp local_settings.py.geoserver.sample local_settings.py
    vim local_settings.py      # edit SITEURL and DATABASES
    cd ..
    python manage.py migrate
    python manage.py runserver 0.0.0.0:8000

I can see a bare GeoNode running on http://localhost:8000. Everything seems fine.

But paver commands do not work:

    $ # set DJANGO_SETTINGS_MODULE, otherwise paver won't use local_settings
    $ export DJANGO_SETTINGS_MODULE=geonode.local_settings

    $ paver sync
    Traceback (most recent call last):
      File "/home/gbassiere/.virtualenvs/geonode_dev/bin/paver", line 8, in <module>
        sys.exit(main())
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/paver/tasks.py", line 890, in main
        _launch_pavement(args)
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/paver/tasks.py", line 858, in _launch_pavement
        exec(compile(source, environment.pavement_file, 'exec'), mod.__dict__)
      File "pavement.py", line 61, in <module>
        from geonode.settings import (
      File "/home/gbassiere/Projets/tuto-geonode/geonode/geonode/settings.py", line 39, in <module>
        from . import serializer
      File "/home/gbassiere/Projets/tuto-geonode/geonode/geonode/serializer.py", line 23, in <module>
        from django.contrib.gis.geos import Polygon
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module>
        from .collections import (  # NOQA
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/geos/collections.py", line 8, in <module>
        from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/geos/geometry.py", line 8, in <module>
        from django.contrib.gis import gdal
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
        from django.contrib.gis.gdal.datasource import DataSource
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/gdal/datasource.py", line 40, in <module>
        from django.contrib.gis.gdal.driver import Driver
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
        from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
        from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/contrib/gis/gdal/libgdal.py", line 15, in <module>
        lib_path = settings.GDAL_LIBRARY_PATH
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/conf/__init__.py", line 82, in __getattr__
        self._setup(name)
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/conf/__init__.py", line 69, in _setup
        self._wrapped = Settings(settings_module)
      File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/conf/__init__.py", line 170, in __init__
        mod = importlib.import_module(self.SETTINGS_MODULE)
      File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "/home/gbassiere/Projets/tuto-geonode/geonode/geonode/local_settings.py", line 94, in <module>
        'GEOSERVER_PUBLIC_HOST', SITE_HOST_NAME
    NameError: name 'SITE_HOST_NAME' is not defined

It seems like there is an import loop because local_settings import settings which import serializer which import the project's settings module (e.g. local_settings). I've been able to use paver after I've edited pavement.py so that settings are imported from local_settings module. I feel like paver should work out-of-the-box, without editing files under version control. Therefore, I must have done something wrong while installing but I basically followed the documentation...

Then I tried to run the test suite, eihter with paver test or python manage.py test or ./test.sh. But I keep on getting errors such as this one:

======================================================================
ERROR: test_remote_document_is_marked_remote (geonode.documents.api.tests.DocumentsApiTests)
Tests creating an external document set its sourcetype to REMOTE.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/test/testcases.py", line 272, in _setup_and_call
    self._pre_setup()
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/test/testcases.py", line 935, in _pre_setup
    self._fixture_setup()
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/test/testcases.py", line 992, in _fixture_setup
    call_command('loaddata', *self.fixtures,
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/modeltranslation/management/commands/loaddata.py", line 49, in handle
    return super().handle(*fixture_labels, **options)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 78, in handle
    self.loaddata(fixture_labels)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 123, in loaddata
    self.load_label(fixture_label)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 190, in load_label
    obj.save(using=self.using)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/modeltranslation/manager.py", line 395, in _update
    return super()._update(values)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
    cursor = self.connection.cursor()
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 236, in _cursor
    with self.wrap_database_errors:
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/gbassiere/.virtualenvs/geonode_dev/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: Problem installing fixture '/home/gbassiere/Projets/tuto-geonode/geonode/geonode/base/fixtures/initial_data.json': connection already closed

Many tests returned this error: connection already closed. What could be wrong with the database? The same database config works fine with runserver.

I'm about to submit a fix for PR #10321 but I wish I could run Django's test suite to validate that my contribution don't break anything.

Regards
Gilles

[mattiagiupponi]

HI @gbassiere

I feel like paver should work out-of-the-box, without editing files under version control. Therefore, I must have done something wrong while installing but I basically followed the documentation...

That's true, paver should work out of the box, feel free to open an issue on it.

Related to the development there are two main choices:

• clone the project and make it run doc
• development with docker doc

The docker way is the easiest to have everything already prepared

I'm about to submit a fix for PR #10321 but I wish I could run Django's test suite to validate that my contribution don't break anything.

We use circleCI to run the whole test suite for each PR. So if something happen dont worry :)

[gbassiere]

Hi Mattia,

Thanks for your guidance.

I think I'm now close to have geonode-core properly configured.

Paver didn't work because of the local_settings.py file. When working directly on geonode-core (e.g. without creating a geonode project), this file isn't necessary because any local parameter can be set in .env file. On that point, the developer doc is misleading.

Unfortunately, I'm still unable to run the test suite :

----------------------------------------------------------------------
Ran 786 tests in 1775.462s

FAILED (failures=13, errors=78)

I get the same results with Docker or without. I don't quite understand what's wrong in my setup. Is it possible that the test suite depends on some parameters in .env file? Please find attached:

• excerpt of the tests output geonode-test-errors.log
• my .env file

I'm aware that CircleCI will check any incoming pull request but I feel like running the test before submitting a PR is the least I could do. Also, it is much easier to fix a breaking test if I can run it on my local instance.

Best regards

[gbassiere]

I took inspiration from CircleCI config to have the test suite running.

For anyone interested, here is how I run tests using Docker:

# Re-build containers if necessary
docker compose --env-file .env_test -f docker-compose-test.yml build --no-cache

# A running GeoNode is necessary (many tests involve HTTP requests to a GeoNode instance)
docker compose --env-file .env_test -f docker-compose-test.yml up -d

# Not sure if this is necessary...
docker compose --env-file .env_test -f docker-compose-test.yml exec db psql -U postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid();'

# Prepare test databases
docker compose --env-file .env_test -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_postgres
docker compose --env-file .env_test -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_geonode
docker compose --env-file .env_test -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_geonode_data
docker compose --env-file .env_test -f docker-compose-test.yml exec db psql -U postgres -d test_geonode -c 'CREATE EXTENSION IF NOT EXISTS postgis;'
docker compose --env-file .env_test -f docker-compose-test.yml exec db psql -U postgres -d test_geonode_data -c 'CREATE EXTENSION IF NOT EXISTS postgis;'

# Load sample data
docker compose --env-file .env_test -f docker-compose-test.yml exec django paver setup_data

# Actually run the whole test suite
docker compose --env-file .env_test -f docker-compose-test.yml exec django bash -c 'coverage run --branch --source=geonode manage.py test -v 3 --keepdb'

# Or run just the tests you want
docker compose --env-file .env_test -f docker-compose-test.yml exec django bash -c 'coverage run --branch --source=geonode manage.py test -v 3 --keepdb geonode.services.tests'

I'm using .env_test and docker-compose-test.yml from master branch, untouched.