Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hosted docker version giving error (NoShardAvailableActionException) #787

Closed
Sunraj751 opened this issue Mar 30, 2024 · 6 comments · Fixed by #793
Closed

Hosted docker version giving error (NoShardAvailableActionException) #787

Sunraj751 opened this issue Mar 30, 2024 · 6 comments · Fixed by #793

Comments

@Sunraj751
Copy link

Sunraj751 commented Mar 30, 2024 •
edited
Loading

Background

  1. I am hosting dockerized version for the latest photon on azure container apps using azure file share.
  2. I am using Azure File Share to persist the data, that is mounted to /photon/photon_data path

Problem

  1. So the problem is after getting the start photon in output (which is after everything is downloaded) it throws an error.
  2. Below is the error log, I have tried re doing the entire process 3 times from scratch but it's still giving this error. Any help would be greatly appreciated🙏 :
2024-03-30T03:04:29.49840  Successfully Connected to container: 'photon-container-app' [Revision: 'photon-container-app--blah-blah', Replica: 'photon-container-app-blah-blah']
Start photon
2024-03-30 03:04:18,427 [main] WARN  org.elasticsearch.node.Node - version [5.6.16-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/photon/photon.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-03-30 03:04:23,618 [main] INFO  de.komoot.photon.elasticsearch.Server - Started elastic search node
2024-03-30 03:04:23,618 [main] INFO  de.komoot.photon.App - Make sure that the ES cluster is ready, this might take some time.
2024-03-30 03:04:53,623 [main] INFO  de.komoot.photon.App - ES cluster is now ready.
Exception in thread "main" NoShardAvailableActionException[No shard available for [get [photon][place][DATABASE_PROPERTIES]: routing [null]]]; nested: RemoteTransportException[[Xlitte1][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:196)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1300(TransportSingleShardAction.java:123)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:252)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1085)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1189)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1167)
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:673)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:660)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: RemoteTransportException[[Xlitte1][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [photon/Ml54BOAPQKuEex9CRZpVww][[photon][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
        at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1104)
        at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:615)
        at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:154)
        at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82)
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:294)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

docker file

FROM openjdk:11-jre

# Install pbzip2 for parallel extraction
RUN apt-get update \
    && apt-get -y install \
        pbzip2 \
        wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /photon
ADD https://github.com/komoot/photon/releases/download/0.5.0/photon-0.5.0.jar /photon/photon.jar
COPY entrypoint.sh ./entrypoint.sh

# Make sure the script is executable
RUN chmod +x ./entrypoint.sh

# Bind external storage to this location 
VOLUME /photon/photon_data
EXPOSE 2322

ENTRYPOINT /photon/entrypoint.sh

entrypoint.sh

#!/bin/bash

# Download elasticsearch index
if [ ! -d "/photon/photon_data/elasticsearch" ]; then
    echo "Downloading search index"

    # Let graphhopper know where the traffic is coming from
    USER_AGENT="docker: sunraj751/photon-geocoder"
    # If you want to install a specific region only, enable the line below and disable the current 'wget' row.
    # Take a look at http://download1.graphhopper.com/public/extracts/by-country-code for your country
    wget --user-agent="$USER_AGENT" -O - http://download1.graphhopper.com/public/extracts/by-country-code/ca/photon-db-ca-latest.tar.bz2 | bzip2 -cd | tar x
fi

# Start photon if elastic index exists
if [ -d "/photon/photon_data/elasticsearch" ]; then
    echo "Start photon"
    java -jar photon.jar $@
else
    echo "Could not start photon, the search index could not be found"
fi
@Sunraj751 Sunraj751 changed the title Hosted version giving errors Hosted docker version giving error (NoShardAvailableActionException) Mar 30, 2024
@Sunraj751
Copy link
Author

Sunraj751 commented Mar 30, 2024 •
edited
Loading

I did try doing as suggested in this discussion post, and now it doesn't work locally on mac ( i.e. updating server.java file): #717.

  1. When i run mvn package command, all the tests are pass and everything is good.
    Screenshot 2024-03-30 at 2 42 59 PM
  2. But when i run the command java -jar photon-*.jar locally I get this error.
    Screenshot 2024-03-30 at 2 43 18 PM

@felix-geovelo
Copy link

I'm facing a similar problem.
I've build my index, exported it to S3.
I've a docker file downloading that index in a persistent volume and launching photon on that index but it always failed to launch with this error :

2024-04-08 14:35:25.493	
Exception in thread "main" NoShardAvailableActionException[No shard available for [get [photon][place][DATABASE_PROPERTIES]: routing [null]]]; nested: RemoteTransportException[[a6JhweV][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
2024-04-08 14:35:25.493	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209)
2024-04-08 14:35:25.494	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:196)
2024-04-08 14:35:25.494	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1300(TransportSingleShardAction.java:123)
2024-04-08 14:35:25.495	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:252)
2024-04-08 14:35:25.495	
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1085)
2024-04-08 14:35:25.496	
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1189)
2024-04-08 14:35:25.496	
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1167)
2024-04-08 14:35:25.497	
	at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:673)
2024-04-08 14:35:25.497	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:660)
2024-04-08 14:35:25.498	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
2024-04-08 14:35:25.499	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2024-04-08 14:35:25.499	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2024-04-08 14:35:25.499	
	at java.base/java.lang.Thread.run(Thread.java:1575)
2024-04-08 14:35:25.500	
Caused by: RemoteTransportException[[a6JhweV][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
2024-04-08 14:35:25.501	
Caused by: [photon/J64ITmmMSB-V5DwgJCJ6UQ][[photon][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
2024-04-08 14:35:25.502	
	at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1104)
2024-04-08 14:35:25.502	
	at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:615)
2024-04-08 14:35:25.502	
	at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:154)
2024-04-08 14:35:25.503	
	at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82)
2024-04-08 14:35:25.503	
	at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
2024-04-08 14:35:25.503	
	at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
2024-04-08 14:35:25.503	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:294)
2024-04-08 14:35:25.504	
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287)
2024-04-08 14:35:25.504	
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
2024-04-08 14:35:25.504	
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
2024-04-08 14:35:25.504	
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662)
2024-04-08 14:35:25.505	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675)
2024-04-08 14:35:25.505	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
2024-04-08 14:35:25.505	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2024-04-08 14:35:25.505	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2024-04-08 14:35:25.505	
	at java.base/java.lang.Thread.run(Thread.java:1575)

Strangely enough, downloading the same index on my personal laptop and launching dockerfile with the volume pointing to my laptop, everything is working like a charm.
I wonder what I am missing here...
Maybe there is a specific way to export a photon index ?
Does just compress the photon_data folder is enough/safe ?

@lonvia
Copy link
Collaborator

lonvia commented Apr 10, 2024

See #717 for a workaround. I suspect we should be doing some additional syncing before shutting down the server after import. It probably works well most of the time and only fails when storage is especially slow for some reason.

@felix-geovelo
Copy link

Thanks for the tip, add the two lines fixes the launching problem for me !

Also, with the persistent volume, performance where horrible (all requests timed-out or 20/30s long)
I've switch from persistent volume to the volume inside my docker and it's know working like a charm !
Thanks !

@lonvia
Copy link
Collaborator

lonvia commented Apr 11, 2024

Good to hear.

Please leave the issue open. This really needs some changes in the code and I don't want to forget again.

@lonvia lonvia mentioned this issue Apr 19, 2024
Merged
@RhinoDevel
Copy link
Contributor

RhinoDevel commented Sep 4, 2024 •
edited
Loading

When using self-compiled version (Elastic search) from commit f55d1df the same error still seems to happen after a restart (maybe, if querying Photon to soon?).

After some seconds though, Photon is usable again (so not that big of a deal..?)!

Happened on Windows, in a VirtualBox VM (without Docker!).

Log output:

Exception in thread "main" NoShardAvailableActionException[No shard available for [get [photon][place][DATABASE_PROPERTIES]: routing [null]]]; nested: RemoteTransportException[[ilgkix0][127.0.0.1:9300][indices:data/read/get[s]]];>
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:196)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1300(TransportSingleShardAction.java:123)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:252)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1085)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1189)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1167)
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:673)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:660)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: RemoteTransportException[[ilgkix0][127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED>
Caused by: [photon/UZskKP1mQgScw7P8lmmd4A][[photon][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
        at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1104)
        at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:615)
        at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:154)
        at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82)
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:294)
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Force a refresh on startup and after updates lonvia/photon
4 participants
@lonvia @RhinoDevel @Sunraj751 @felix-geovelo