Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WARNING: Error from ctags: ctags: Warning: cannot open input file: No such file or directory #3342

Closed
totoroliu opened this issue Oct 30, 2020 · 19 comments
Closed

WARNING: Error from ctags: ctags: Warning: cannot open input file: No such file or directory #3342

totoroliu opened this issue Oct 30, 2020 · 19 comments
Labels
indexer question

Comments

@totoroliu
Copy link

On Ubuntu 18.04 machine, I just upgrade:
Opengrok: from v1.3.11 to v1.5.5
ctags: from /usr/local/bin/ctags v5.8 (e87283aa) Mar 30 2020 to /snap/bin/universal-ctags v5.8 (e852ee0e) Oct 7 2020
java: from java-1.8.0-openjdk-amd64 to java-1.11.0-openjdk-amd64
Tomcat: 9.0.16-3ubuntu0.18.04.1
Git: 2.17.1

Then,
when I run index update:
#!/bin/bash -x
time java -server -Xms20G -Xmx20G \
-Djava.util.logging.config.file=/opengrok/etc/logging.properties \
-jar /opengrok/dist/lib/opengrok.jar \
-c /snap/bin/universal-ctags \
-s /opengrok/src -d /opengrok/data -H -P -S -G \
-v --progress \
--threads 6 --historyThreads 6 -m 15360 \
-W /opengrok/etc/configuration.xml -U http://localhost:8080/source

I got tons of WARNINGs (probably one WARNING per one source file under /opengrok/src)
about ctags cannot open input file (No such file or directory):
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_efuse.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.series" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Rega-Sense" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.soc" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_event.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/linker.ld" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Video2000" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/soc.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/sound/firewire/oxfw/oxfw.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_fifoctrl_bitdef.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/fs/hfs/mdb.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Zuerich-upc-cablecom" : No such file or directory

but when I check the file path, the file does exist there:
-rw-rw-r-- 1 devops devops 21092 Jun 2 20:05 /opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c
-rw-rw-r-- 1 devops devops 1614 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_efuse.h
-rw-rw-r-- 1 devops devops 2359 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_event.h
-rw-rw-r-- 1 devops devops 3269 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_fifoctrl_bitdef.h
-rw-rw-r-- 1 devops devops 10371 Aug 13 20:05 /opengrok/src/Yocto-Valkyrie/kernel/fs/hfs/mdb.c
-rw-rw-r-- 1 devops devops 10303 Apr 8 2020 /opengrok/src/Yocto-Valkyrie/kernel/sound/firewire/oxfw/oxfw.c
-rwxrwxr-x 1 devops devops 3733 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Rega-Sense
-rwxrwxr-x 1 devops devops 218 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Video2000
-rwxrwxr-x 1 devops devops 201 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Zuerich-upc-cablecom
-rw-rw-r-- 1 devops devops 257 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.series
-rw-rw-r-- 1 devops devops 295 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.soc
-rw-rw-r-- 1 devops devops 141 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/linker.ld
-rw-rw-r-- 1 devops devops 3010 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/soc.h

Are these ctags ERROR (opengrok WARNING) correct?
How to debug or fix these these ERROR/WARNING?
Does opengrok index update break?
@vladak
Copy link
Member

vladak commented Oct 30, 2020 •
edited
Loading

Is there anything interesting in indexer logs ? (other than the errors from ctags)

Firstly, upgrade only one thing at a time. Try upgrading just Java and OpenGrok first. See if it breaks the indexer. Then upgrade Universal ctags.

@vladak
Copy link
Member

vladak commented Oct 30, 2020

To get more info of what is happening run the indexer with --threads 1 so that only single ctags process is created. Then strace both the ctags process and the indexer process to see what is going on w.r.t. I/O syscalls.

Also, what is /opengrok/src exactly ?

@vladak
Copy link
Member

vladak commented Oct 30, 2020

Also, does this happen with all projects under source root or just some of them ?

@totoroliu
Copy link
Author

opengrok18.0.log.gz
2020-10-30 11:27:14.918-0700 WARNING t10324 Ctags.lambda$run$0: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c" : No such file or directory
2020-10-30 11:27:14.945-0700 FINEST t10314 Statistics.logIt: file '/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c' indexed (took 33 ms)

$ tree -L 2 /opengrok/src/
/opengrok/src/
├── Creole
│   └── creole
├── NetXtreme
│   └── netxtreme
├── Trunk:MxC
│   ├── brcm
│   ├── ing
│   ├── map
│   ├── meego
│   ├── meego.git
│   ├── mirrors
│   ├── opensource
│   ├── os
│   ├── qa
│   ├── repo_list
│   └── tizen
└── Yocto-Valkyrie
├── arm-trusted-firmware
├── bcm-secimage
├── ffmpeg
├── halapis
├── kernel
├── manifest
├── meta-brcm
├── meta-openembedded
├── meta-raspberrypi
├── poky
├── qemu
├── sbi
├── sec-keys
├── vkcmodel
├── vkfirmware
├── vkh264dec
├── vkil
├── vkipc
├── vkservices
├── vksim
├── vktest
├── vkutils
└── zephyr

@totoroliu
Copy link
Author

totoroliu commented Oct 30, 2020 •
edited
Loading

it looks like happen to all the files under source root for all projects
I got 19 log file after running the re-indexing for all of 4 projects
The gzip tarball of all of 19 log files is 58MB.

@totoroliu
Copy link
Author

opengrok19.0.log.gz
this log has ERROR for "/opengrok/src/Creole/creole/ project:

@totoroliu
Copy link
Author

opengrok7.0.log.gz
This log has Error for /opengrok/src/Trunk:MxC/ and /opengrok/src/NetXtreme/netxtreme/

@totoroliu
Copy link
Author

I'll try --threads 1 now.

@vladak
Copy link
Member

vladak commented Oct 30, 2020

$ tree -L 2 /opengrok/src/
/opengrok/src/
├── Creole

I meant - is there something special about the file system holding this data ?

@totoroliu
Copy link
Author

The root / filesystem is on LVM
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/lvm-root 148G 83G 59G 59% /

devops@vpx-opengrok:/opengrok$ ls -l /opengrok/src
total 16
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 Creole
drwxrwxr-x 3 devops devops 4096 Oct 7 11:16 NetXtreme
drwxrwxr-x 12 devops devops 4096 Mar 30 2020 Trunk:MxC
drwxrwxr-x 25 devops devops 4096 Mar 30 2020 Yocto-Valkyrie
devops@vpx-opengrok:/opengrok$ ls -l /opengrok/src/*
/opengrok/src/Creole:
total 4
drwxrwxr-x 18 devops devops 4096 Oct 30 16:17 creole

/opengrok/src/NetXtreme:
total 4
drwxrwxr-x 5 devops devops 4096 Oct 22 00:00 netxtreme

'/opengrok/src/Trunk:MxC':
total 108
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 brcm
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 ing
drwxrwxr-x 120 devops devops 4096 Mar 30 2020 map
drwxrwxr-x 453 devops devops 16384 Mar 30 2020 meego
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 meego.git
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 mirrors
drwxrwxr-x 101 devops devops 4096 Mar 30 2020 opensource
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 os
drwxrwxr-x 17 devops devops 4096 Mar 30 2020 qa
-rw-rw-r-- 1 devops devops 53876 Mar 30 2020 repo_list
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 tizen

/opengrok/src/Yocto-Valkyrie:
total 92
drwxrwxr-x 20 devops devops 4096 Oct 14 10:05 arm-trusted-firmware
drwxrwxr-x 6 devops devops 4096 Apr 18 2020 bcm-secimage
drwxrwxr-x 19 devops devops 4096 Sep 25 04:05 ffmpeg
drwxrwxr-x 28 devops devops 4096 Aug 21 20:05 halapis
drwxrwxr-x 26 devops devops 4096 Aug 13 20:05 kernel
drwxrwxr-x 3 devops devops 4096 Sep 24 20:05 manifest
drwxrwxr-x 15 devops devops 4096 Apr 2 2020 meta-brcm
drwxrwxr-x 14 devops devops 4096 Apr 30 2020 meta-openembedded
drwxrwxr-x 18 devops devops 4096 Mar 30 2020 meta-raspberrypi
drwxrwxr-x 12 devops devops 4096 Apr 3 2020 poky
drwxrwxr-x 45 devops devops 4096 Mar 30 2020 qemu
drwxrwxr-x 11 devops devops 4096 Aug 7 20:05 sbi
drwxrwxr-x 5 devops devops 4096 Sep 17 20:05 sec-keys
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 vkcmodel
drwxrwxr-x 8 devops devops 4096 Aug 4 20:05 vkfirmware
drwxrwxr-x 7 devops devops 4096 Jul 1 04:05 vkh264dec
drwxrwxr-x 8 devops devops 4096 Mar 30 2020 vkil
drwxrwxr-x 9 devops devops 4096 Jul 24 20:05 vkipc
drwxrwxr-x 6 devops devops 4096 Mar 30 2020 vkservices
drwxrwxr-x 8 devops devops 4096 Sep 30 04:05 vksim
drwxrwxr-x 9 devops devops 4096 Oct 8 04:05 vktest
drwxrwxr-x 11 devops devops 4096 Oct 29 04:05 vkutils
drwxrwxr-x 19 devops devops 4096 May 8 18:53 zephyr

@totoroliu
Copy link
Author

opengrok0.0.log.gz
this is the log for --threads 1:

#!/bin/bash -x
time java -server -Xms20G -Xmx20G \
    -Djava.util.logging.config.file=/opengrok/etc/logging.properties \
    -jar /opengrok/dist/lib/opengrok.jar \
    -c /snap/bin/universal-ctags \
    -s /opengrok/src -d /opengrok/data -H -P -S -G \
    -v --progress \
    --threads 1 --historyThreads 6 -m 15360 \
    -W /opengrok/etc/configuration.xml -U http://localhost:8080/source

@vladak
Copy link
Member

vladak commented Oct 31, 2020

There's nothing interesting in the log from what I can tell. You need to run strace in the ctags program while it is being executed by the indexer to see more.

@totoroliu
Copy link
Author

how to run strace in ctags program while it's been executed by indexer?
do you have instruction?

@vladak
Copy link
Member

vladak commented Nov 1, 2020 •
edited
Loading

Basically, you run the indexer, wait for the ctags process to be spawned and then run strace on it. strace can trace only one process at a time so it is good to restrict the indexer to run only one ctags process (otherwise you would have to run multiple strace instances to get data from all the processes) with the -T/--threads option.

@Fire30
Copy link

Fire30 commented Nov 3, 2020

I also was having this issue. I believe it is due to the snap release of ctags. If I instead use a locally compiled ctags that I have gathered from https://github.com/universal-ctags/ctags/ it works fine.

@vladak
Copy link
Member

vladak commented Nov 3, 2020

It would be nice if you could collect more data and file a issue under https://github.com/universal-ctags/ctags (assuming this is not caused by the way how the snap package is constructed).

@Fire30
Copy link

Fire30 commented Nov 3, 2020

It is due to snap packaging I have determined. Reading the page https://snapcraft.io/universal-ctags

Compared to universal-ctags installed using apt, or compiled from source, this snap currently has some limitations, introduced by snap security restrictions:

Source files cannot be scanned if they are:

Outside the user's $HOME directory, or
In a top-level hidden directory, eg. ~/.src.
However, it can scan source in lower-level hidden directories, eg. ~/proj/.src.

@totoroliu
Copy link
Author

Thank you for the info.
I'll remove snap universal-ctags and change to build from source.

@shareef1973
Copy link

uninstalling Snap Ctags , then installed it using APT , i cannot reproduce the problem any more, after doing that
so this is certainly related to the Snap ctags

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
indexer question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@totoroliu @Fire30 @vladak @shareef1973