Problem running in container for directories without any version control

[heldbrendel]

We have multiple directories with source code that does not belong into a version control system as it is only as a reference.
When trying to use OpenGrok in a Docker container to index those directories the indexer runs into a loop.

The problem can easily be recreated by using any git repository and removing the .git folder.
For this case I used hellogitworld.git.

OpenGrok somehow identifies the folder as using Perforce.

These are the log messages:

opengrok  | 2023-08-28 10:51:17,418     INFO opengrok_tools | Running version Warning: JAVA_HOME environment variable is not set.
opengrok  | 1.12.14
opengrok  | 2023-08-28 10:51:17,419    DEBUG opengrok_tools | URL_ROOT = source
opengrok  | 2023-08-28 10:51:17,419    DEBUG opengrok_tools | URI = http://localhost:8080/source/
opengrok  | 2023-08-28 10:51:17,419     INFO opengrok_tools | synchronization period = 60 minutes
opengrok  | 2023-08-28 10:51:17,419     INFO opengrok_tools | Deploying web application
opengrok  | 2023-08-28 10:51:17,420    DEBUG opengrok_tools | Repacking /opengrok/lib/source.war with custom configuration path to /tmp/OpenGroktmpWar3m7c8010.war
opengrok  | 2023-08-28 10:51:18,257    DEBUG opengrok_tools | Performing substitution of '/var/opengrok/etc/configuration.xml' with '/opengrok/etc/configuration.xml'
opengrok  | 2023-08-28 10:51:19,170    DEBUG opengrok_tools | Installing '/tmp/OpenGroktmpWar3m7c8010.war' to '/usr/local/tomcat/webapps/source.war'
opengrok  | 2023-08-28 10:51:19,192    DEBUG opengrok_tools | Extra environment: {'OPENGROK_NO_MIRROR': 'true'}
opengrok  | 2023-08-28 10:51:19,193     INFO opengrok_tools | Creating bare configuration in /opengrok/etc/configuration.xml
opengrok  | 2023-08-28 10:51:19,193    DEBUG opengrok_tools | hg = /usr/bin/hg
opengrok  | 2023-08-28 10:51:19,193    DEBUG opengrok_tools | cvs = /usr/bin/cvs
opengrok  | 2023-08-28 10:51:19,194    DEBUG opengrok_tools | svn = /usr/bin/svn
opengrok  | 2023-08-28 10:51:19,194    DEBUG opengrok_tools | sccs = /usr/bin/sccs
opengrok  | 2023-08-28 10:51:19,194    DEBUG opengrok_tools | p4 = /usr/bin/p4
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | blame = /usr/bin/blame
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | bzr = /usr/bin/bzr
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | Java options: ['-Dorg.opengrok.indexer.history.Subversion=/usr/bin/svn', '-Dorg.opengrok.indexer.history.cvs=/usr/bin/cvs', '-Dorg.opengrok.indexer.history.SCCS=/usr/bin/sccs', '-Dorg.opengrok.indexer.history.RCS=/usr/bin/blame', '-Dorg.opengrok.indexer.history.Bazaar=/usr/bin/bzr', '-Dorg.opengrok.indexer.history.Mercurial=/usr/bin/hg', '-Dorg.opengrok.indexer.history.Perforce=/usr/bin/p4']
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | Could not detemine Java home using standard means, trying JAVA_HOME: /opt/java/openjdk
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | Java = /opt/java/openjdk/bin/java
opengrok  | 2023-08-28 10:51:19,195    DEBUG opengrok_tools | Java command: ['/opt/java/openjdk/bin/java', '-Dorg.opengrok.indexer.history.Subversion=/usr/bin/svn', '-Dorg.opengrok.indexer.history.cvs=/usr/bin/cvs', '-Dorg.opengrok.indexer.history.SCCS=/usr/bin/sccs', '-Dorg.opengrok.indexer.history.RCS=/usr/bin/blame', '-Dorg.opengrok.indexer.history.Bazaar=/usr/bin/bzr', '-Dorg.opengrok.indexer.history.Mercurial=/usr/bin/hg', '-Dorg.opengrok.indexer.history.Perforce=/usr/bin/p4', '-jar', '/opengrok/lib/opengrok.jar', '-s', '/opengrok/src', '-d', '/opengrok/data', '-c', '/usr/local/bin/ctags', '--remote', 'on', '-H', '-S', '-W', '/opengrok/etc/configuration.xml', '--noIndex']
opengrok  | 2023-08-28 10:51:19,196    DEBUG opengrok_tools | working directory = /usr/local/tomcat
opengrok  | 2023-08-28 10:51:19,196    DEBUG opengrok_tools | command = '/opt/java/openjdk/bin/java -Dorg.opengrok.indexer.history.Subversion=/usr/bin/svn -Dorg.opengrok.indexer.history.cvs=/usr/bin/cvs -Dorg.opengrok.indexer.history.SCCS=/usr/bin/sccs -Dorg.opengrok.indexer.history.RCS=/usr/bin/blame -Dorg.opengrok.indexer.history.Bazaar=/usr/bin/bzr -Dorg.opengrok.indexer.history.Mercurial=/usr/bin/hg -Dorg.opengrok.indexer.history.Perforce=/usr/bin/p4 -jar /opengrok/lib/opengrok.jar -s /opengrok/src -d /opengrok/data -c /usr/local/bin/ctags --remote on -H -S -W /opengrok/etc/configuration.xml --noIndex'
opengrok  | 2023-08-28 10:51:19,196    DEBUG opengrok_tools | Waiting for process with PID 7
opengrok  | 2023-08-28 10:51:19,431     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.index.Indexer parseOptions
opengrok  | 2023-08-28 10:51:19,431     INFO opengrok_tools | INFO: Indexer options: [-s, /opengrok/src, -d, /opengrok/data, -c, /usr/local/bin/ctags, --remote, on, -H, -S, -W, /opengrok/etc/configuration.xml, --noIndex]
opengrok  | 2023-08-28 10:51:19,558     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.index.Indexer main
opengrok  | 2023-08-28 10:51:19,558     INFO opengrok_tools | INFO: Indexer version 1.12.14 (f412b9bfd313ef5f78ac71904deb2e9209f48aed) running on Java 11.0.20
opengrok  | 2023-08-28 10:51:19,649     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.asax.Asax.ASAX" in "langmap" option
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.ascx.Ascx.ASCX" in "langmap" option
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.aspx.Aspx.ASPX" in "langmap" option
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,650     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.htm.Htm.HTM" in "langmap" option
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.html.Html.HTML" in "langmap" option
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.master.Master.MASTER" in "langmap" option
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.xaml.Xaml.XAML" in "langmap" option
opengrok  | 2023-08-28 10:51:19,651     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.analysis.Ctags lambda$run$0
opengrok  | 2023-08-28 10:51:19,652     INFO opengrok_tools | WARNING: Error from ctags: ctags: Warning: Unknown language "XML:+.xml.Xml.XML" in "langmap" option
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.configuration.RuntimeEnvironment validateUniversalCtags
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools | INFO: Using ctags: Universal Ctags 6.0.0(d104ee5b), Copyright (C) 2015-2022 Universal Ctags Team
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools | Universal Ctags is derived from Exuberant Ctags.
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools | Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools |   Compiled: Aug 22 2023, 12:58:48
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools |   URL: https://ctags.io/
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools |   Output version: 0.0
opengrok  | 2023-08-28 10:51:19,673     INFO opengrok_tools |   Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +packcc, +optscript
opengrok  | 2023-08-28 10:51:19,674     INFO opengrok_tools | Aug 28, 2023 10:51:19 AM org.opengrok.indexer.index.Indexer prepareIndexer
opengrok  | 2023-08-28 10:51:19,674     INFO opengrok_tools | INFO: Scanning for repositories in [/opengrok/src/hellogitworld] (down to 3 levels below source root)
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools | Aug 28, 2023 10:51:27 AM org.opengrok.indexer.util.Executor exec
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools | WARNING: Non-zero exit status 1 from command [/usr/bin/p4 dirs $'*'] in directory '/opengrok/src/hellogitworld': Perforce client error:
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools |   Connect to server failed; check $P4PORT.
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools |   TCP connect to perforce:1666 failed.
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools |   Temporary failure in name resolution
opengrok  | 2023-08-28 10:51:27,717     INFO opengrok_tools |
opengrok  | 2023-08-28 10:51:35,728     INFO opengrok_tools | Aug 28, 2023 10:51:35 AM org.opengrok.indexer.util.Executor exec
opengrok  | 2023-08-28 10:51:35,728     INFO opengrok_tools | WARNING: Non-zero exit status 1 from command [/usr/bin/p4 files $'*'] in directory '/opengrok/src/hellogitworld': Perforce client error:
opengrok  | 2023-08-28 10:51:35,729     INFO opengrok_tools |   Connect to server failed; check $P4PORT.
opengrok  | 2023-08-28 10:51:35,729     INFO opengrok_tools |   TCP connect to perforce:1666 failed.
opengrok  | 2023-08-28 10:51:35,729     INFO opengrok_tools |   Temporary failure in name resolution
opengrok  | 2023-08-28 10:51:35,729     INFO opengrok_tools |
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools | Aug 28, 2023 10:51:43 AM org.opengrok.indexer.util.Executor exec
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools | WARNING: Non-zero exit status 1 from command [/usr/bin/p4 dirs $'*'] in directory '/opengrok/src/hellogitworld/resources': Perforce client error:
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools |   Connect to server failed; check $P4PORT.
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools |   TCP connect to perforce:1666 failed.
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools |   Temporary failure in name resolution
opengrok  | 2023-08-28 10:51:43,778     INFO opengrok_tools |

I tried multiple ways to disable the mirroring and or not handling the directories as projects by using the following settings:

docker-compose.yml:

version: '3.9'
services:
  opengrok:
    image: opengrok/docker:latest
    container_name: opengrok
    restart: always    
    ports:
      - "8080:8080/tcp"
    environment:
      - OPENGROK_LOG_LEVEL=DEBUG
      - URL_ROOT=source
      - AVOID_PROJECTS=true
      - NOMIRROR=true
    volumes:
       - '.\src\:/opengrok/src/'    
       - '.\etc\:/opengrok/etc/'    
       - '.\data\:/opengrok/data/' 

mirror.yml:

projects:
  hellogitworld:
    disabled: true
    disabled-reason: "not a git project"
    ignore_errors: true

I would be really grateful if someone could point me in the right direction on how to index directories without any version control.

[vladak]

I tried to replicate with 1.12.14 but could not. The contents of mirror.yml are not relevant because with NOMIRROR=true the mirroring is not done.

The failed Perforce commands is just a red herring. These are run by the indexer to detect what kind of repository a directory is, it is opportunistic check.

If this was not 1.12.14 then I'd say you are bumping into #4374, because the last command in the log was the indexer run with the --noIndex option, however that was fixed in 1.12.11.

[vladak]

If you list running processes in the container, what does the output look like ?

[heldbrendel]

I will check the behaviour again.

In the meantime here is the output of docker top opengrok:

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                8694                8673                2                   14:48               ?                   00:00:03            python3 /scripts/start.py
root                8940                8694                35                  14:49               ?                   00:00:25            /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/opengrok/data/tomcat_temp org.apache.catalina.startup.Bootstrap start

[vladak]

There is just the entry point program and Tomcat; not a sight of hung indexer.

[heldbrendel]

You are right.

I just tried again and after waiting a bit more and ignoring the Perforce warnings the directory was successfully indexed and searchable.

Thank you very much for your time.