opensearch version of photon does not start up

[metaminimalist]

Today, I tried to start photon with opensearch from https://github.com/komoot/photon/releases/download/0.6.2/photon-opensearch-0.6.2.jar with the latest photon database from https://download1.graphhopper.com/public/photon-db-latest.tar.bz2 (downloaded 24 Jan 2025)

Unfortunately, it crashed, immediately after the ES cluster is now ready. message:

% /usr/lib/jvm/java-21-openjdk-amd64/bin/java -jar /var/lib/komoot-photon/runtime/photon-opensearch-0.6.2.jar
Cluster Name: photon
Base Path:    /var/lib/komoot-photon/./photon_data
Num Of Node:  1
Node Name:      Node 1
HTTP Port:      9201
Data Directory: /var/lib/komoot-photon/./photon_data/node_1/data
Log Directory:  /var/lib/komoot-photon/./photon_data/node_1/logs
[2025-01-24T16:08:20,725][INFO ][o.o.n.Node               ] version[2.18.0], pid[1591666], build[unknown/unknown/unknown], OS[Linux/5.15.0-130-generic/amd64], JVM[Ubuntu/OpenJDK 64-B
it Server VM/21.0.5/21.0.5+11-Ubuntu-1ubuntu122.04]
[2025-01-24T16:08:20,727][INFO ][o.o.n.Node               ] JVM home [/usr/lib/jvm/java-21-openjdk-amd64]
[2025-01-24T16:08:20,729][INFO ][o.o.n.Node               ] JVM arguments []
[2025-01-24T16:08:20,735][INFO ][o.o.p.PluginsService     ] no modules loaded
[2025-01-24T16:08:20,736][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.analysis.common.CommonAnalysisPlugin]
[2025-01-24T16:08:20,736][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.transport.Netty4Plugin]
[2025-01-24T16:08:20,761][WARN ][stderr                   ] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-01-24T16:08:20,761][WARN ][stderr                   ] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-01-24T16:08:20,762][WARN ][stderr                   ] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-01-24T16:08:20,798][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2025-01-24T16:08:20,800][INFO ][o.o.e.NodeEnvironment    ] using [1] data paths, mounts [[/ (/dev/md2)]], net usable_space [1.1tb], net total_space [1.9tb], types [ext4]
[2025-01-24T16:08:20,803][INFO ][o.o.e.NodeEnvironment    ] heap size [29.5gb], compressed ordinary object pointers [true]
[2025-01-24T16:08:20,836][WARN ][o.a.l.i.v.VectorizationProvider] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
[2025-01-24T16:08:20,886][INFO ][o.o.n.Node               ] node name [Node 1], node ID [6X0TcXzFSNefejzYtxyHng], cluster name [photon], roles [data, cluster_manager]
[2025-01-24T16:08:20,903][INFO ][o.o.e.ExtensionsManager  ] ExtensionsManager initialized
[2025-01-24T16:08:21,572][INFO ][o.o.t.NettyAllocator     ] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_al[34/1868$size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2025-01-24T16:08:21,628][INFO ][o.o.d.DiscoveryModule    ] using discovery type [single-node] and seed hosts providers [settings]
[2025-01-24T16:08:21,760][WARN ][o.o.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-01-24T16:08:21,873][INFO ][o.o.n.Node               ] initialized
[2025-01-24T16:08:21,873][INFO ][o.o.n.Node               ] starting ...
[2025-01-24T16:08:21,926][INFO ][o.o.t.TransportService   ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2025-01-24T16:08:21,989][INFO ][o.o.c.c.Coordinator      ] cluster UUID [YvovQNZuRPqHLO7UwMJa4w]
[2025-01-24T16:08:22,060][INFO ][o.o.c.s.MasterService    ] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 2, version: 3, delta: cluster-manager node changed {previous [], current [{Node 1}{6X0TcXzFSNefejzYtxyHng}{u9r2xch_QXegrd7Uwa73RA}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}
[2025-01-24T16:08:22,091][INFO ][o.o.c.s.ClusterApplierService] cluster-manager node changed {previous [], current [{Node 1}{6X0TcXzFSNefejzYtxyHng}{u9r2xch_QXegrd7Uwa73RA}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}, term: 2, version: 3, reason: Publication{term=2, version=3}
[2025-01-24T16:08:22,118][INFO ][o.o.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[2025-01-24T16:08:22,118][INFO ][o.o.n.Node               ] started
[2025-01-24T16:08:22,118][INFO ][o.o.d.PeerFinder         ] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Node 1}{6X0TcXzFSNefejzYtxyHng}{u9r2xch_QXegrd7Uwa73RA}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]
[2025-01-24T16:08:22,121][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-01-24T16:08:22,127][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-01-24T16:08:22,133][INFO ][o.o.g.GatewayService     ] recovered [0] indices into cluster_state
[2025-01-24T16:08:22,285][INFO ][d.k.p.App                ] Make sure that the ES cluster is ready, this might take some time.
[2025-01-24T16:08:22,389][INFO ][d.k.p.App                ] ES cluster is now ready.
[2025-01-24T16:08:22,408][WARN ][stderr                   ] Exception in thread "main" org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [index_not_found_exception] no such index [photon]
[2025-01-24T16:08:22,408][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.extractAndWrapCause(ApacheHttpClient5Transport.java:1153)
[2025-01-24T16:08:22,408][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.performRequest(ApacheHttpClient5Transport.java:158)
[2025-01-24T16:08:22,408][WARN ][stderr                   ]     at org.opensearch.client.opensearch.OpenSearchClient.get(OpenSearchClient.java:742)
[2025-01-24T16:08:22,408][WARN ][stderr                   ]     at org.opensearch.client.opensearch.OpenSearchClient.get(OpenSearchClient.java:758)
[2025-01-24T16:08:22,408][WARN ][stderr                   ]     at de.komoot.photon.Server.loadFromDatabase(Server.java:159)
[2025-01-24T16:08:22,409][WARN ][stderr                   ]     at de.komoot.photon.Server.updateIndexSettings(Server.java:134)
[2025-01-24T16:08:22,409][WARN ][stderr                   ]     at de.komoot.photon.App.main(App.java:55)
[2025-01-24T16:08:22,409][WARN ][stderr                   ] Caused by: org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [index_not_found_exception] no such index [photon]
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at org.opensearch.client.transport.Endpoint.exceptionConverter(Endpoint.java:102)
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.prepareResponse(ApacheHttpClient5Transport.java:547)
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.lambda$performRequestAsync$0(ApacheHttpClient5Transport.java:191)
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-01-24T16:08:22,410][WARN ][stderr                   ]     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport$1.completed(ApacheHttpClient5Transport.java:236)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport$1.completed(ApacheHttpClient5Transport.java:225)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.apache.hc.core5.concurrent.BasicFuture.completed(BasicFuture.java:148)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.apache.hc.core5.concurrent.ComplexFuture.completed(ComplexFuture.java:72)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2$1.completed(InternalAbstractHttpAsyncClient.java:310)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer$1.completed(AbstractAsyncResponseConsumer.java:101)
[2025-01-24T16:08:22,411][WARN ][stderr                   ]     at org.apache.hc.core5.http.nio.entity.AbstractBinAsyncEntityConsumer.completed(AbstractBinAsyncEntityConsumer.java:87)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.streamEnd(AbstractBinDataConsumer.java:83)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer.streamEnd(AbstractAsyncResponseConsumer.java:142)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.streamEnd(HttpAsyncMainClientExec.java:284)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.dataEnd(ClientHttp1StreamHandler.java:276)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.dataEnd(ClientHttp1StreamDuplexer.java:366)
[2025-01-24T16:08:22,412][WARN ][stderr                   ]     at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:338)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:143)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:176)
[2025-01-24T16:08:22,413][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:125)
[2025-01-24T16:08:22,414][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:92)
[2025-01-24T16:08:22,414][WARN ][stderr                   ]     at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
[2025-01-24T16:08:22,414][WARN ][stderr                   ]     at java.base/java.lang.Thread.run(Thread.java:1583)

I had expected it to start up normally and serve api requests.

For now, I'm using the normal version without opensearch, which works as expected.

[mtmail]

I'd guess the data for the experimental opensearch implementation is in https://download1.graphhopper.com/public/experimental/

[travelr]

[2025-02-01T15:19:43,265][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:19:43,269][INFO ][o.o.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9202}, bound_addresses {[::1]:9202}, {127.0.0.1:9202}
[2025-02-01T15:19:43,269][INFO ][o.o.n.Node               ] started
[2025-02-01T15:19:43,271][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:19:43,671][INFO ][o.o.g.GatewayService     ] recovered [1] indices into cluster_state
[2025-02-01T15:19:45,216][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:19:45,587][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:19:45,607][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:19:45,781][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/bcm0QVUQS32nnP338yKE2w]
[2025-02-01T15:19:45,980][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-01T15:20:13,386][WARN ][stderr                   ] Exception in thread "main" org.codelibs.opensearch.runner.OpenSearchRunnerException: ensureYellow timed out, cluster state:
[2025-02-01T15:20:14,021][WARN ][stderr                   ] cluster uuid: fPEqZ1zTQCKSwBfHJFMjSA [committed: true]
[2025-02-01T15:20:14,021][WARN ][stderr                   ] version: 30
[2025-02-01T15:20:14,021][WARN ][stderr                   ] state uuid: YKOlgABAQ7WnD9u3XN06xQ
[2025-02-01T15:20:14,021][WARN ][stderr                   ] from_diff: false
[2025-02-01T15:20:14,021][WARN ][stderr                   ] meta data version: 8
[2025-02-01T15:20:14,021][WARN ][stderr                   ]    coordination_metadata:
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       term: 6
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       last_committed_config: VotingConfiguration{gq9NDPFTQra4lXVtyzVxsQ}
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       last_accepted_config: VotingConfiguration{gq9NDPFTQra4lXVtyzVxsQ}
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       voting tombstones: []
[2025-02-01T15:20:14,021][WARN ][stderr                   ]    [photon/bcm0QVUQS32nnP338yKE2w]: v[23], mv[2], sv[3], av[1]
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       0: p_term [5], isa_ids [AexQzb_dS5-td1SabJtGuA]
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       1: p_term [4], isa_ids [DiaUby1lS_S4fOeEIhzTng]
[2025-02-01T15:20:14,021][WARN ][stderr                   ]       2: p_term [6], isa_ids [p1N5duX3Smmcmh1wUbO7eA]
[2025-02-01T15:20:14,022][WARN ][stderr                   ]       3: p_term [5], isa_ids [1T7kLP63Tf2oUIZUL7bGqQ]
[2025-02-01T15:20:14,022][WARN ][stderr                   ]       4: p_term [5], isa_ids [iLV9HNl2S3qokTMSv70i9A]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] metadata customs:
[2025-02-01T15:20:14,022][WARN ][stderr                   ]    index-graveyard: IndexGraveyard[[]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] nodes:
[2025-02-01T15:20:14,022][WARN ][stderr                   ]    {Node 1}{gq9NDPFTQra4lXVtyzVxsQ}{gVBpUuicTW6dyiFB_fc6oA}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}, local, cluster-manager
[2025-02-01T15:20:14,022][WARN ][stderr                   ] routing_table (version 3):
[2025-02-01T15:20:14,022][WARN ][stderr                   ] -- index [[photon/bcm0QVUQS32nnP338yKE2w]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] ----shard_id [photon][0]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][0], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=AexQzb_dS5-td1SabJtGuA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] ----shard_id [photon][1]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][1], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=DiaUby1lS_S4fOeEIhzTng], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] ----shard_id [photon][2]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][2], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=p1N5duX3Smmcmh1wUbO7eA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,022][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] ----shard_id [photon][3]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] --------[photon][3], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[deciders_throttled]]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] ----shard_id [photon][4]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] --------[photon][4], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=iLV9HNl2S3qokTMSv70i9A], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,095][WARN ][stderr                   ] routing_nodes:
[2025-02-01T15:20:14,095][WARN ][stderr                   ] -----node_id[gq9NDPFTQra4lXVtyzVxsQ][V]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][0], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=AexQzb_dS5-td1SabJtGuA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][1], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=DiaUby1lS_S4fOeEIhzTng], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][2], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=p1N5duX3Smmcmh1wUbO7eA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][4], node[gq9NDPFTQra4lXVtyzVxsQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=iLV9HNl2S3qokTMSv70i9A], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] ---- unassigned
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][3], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[deciders_throttled]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-02-01T20:19:43.269Z], delayed=false, allocation_status[no_attempt]]
[2025-02-01T15:20:14,096][WARN ][stderr                   ] tasks: (0):
[2025-02-01T15:20:14,096][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.onFailure(OpenSearchRunner.java:1101)
[2025-02-01T15:20:14,096][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.ensureYellow(OpenSearchRunner.java:758)
[2025-02-01T15:20:14,096][WARN ][stderr                   ] 	at de.komoot.photon.Server.startInternal(Server.java:81)
[2025-02-01T15:20:14,096][WARN ][stderr                   ] 	at de.komoot.photon.Server.start(Server.java:40)
[2025-02-01T15:20:14,096][WARN ][stderr                   ] 	at de.komoot.photon.App.main(App.java:41)

With the experimental dataset the full one is still failing. Loading a small dataset e.g. Ireland works without a problem.
Increasing memory size also does not help java -Xmx4g -jar photon-opensearch-0.6.2.jar

[lonvia]

I've just tested the latest planet extract and it starts up fine for me. You might be running out of disk space or something similar. Are there any other error messages in the output before the part you posted?

[metaminimalist]

ok, so I loaded down the datafile from https://download1.graphhopper.com/public/experimental/ and I started photon-opensearch like this:

/usr/lib/jvm/java-21-openjdk-amd64/bin/java -jar /var/lib/komoot-photon/runtime/photon-opensearch-0.6.2.jar -structured

It started up without exploding.

% /usr/lib/jvm/java-21-openjdk-amd64/bin/java -jar /var/lib/komoot-photon/runtime/photon-opensearch-0.6.2.jar -structured

Cluster Name: photon
Base Path:    /var/lib/komoot-photon/./photon_data
Num Of Node:  1
Node Name:      Node 1
HTTP Port:      9201
Data Directory: /var/lib/komoot-photon/./photon_data/node_1/data
Log Directory:  /var/lib/komoot-photon/./photon_data/node_1/logs
[2025-02-12T16:30:53,271][INFO ][o.o.n.Node               ] version[2.18.0], pid[3263246], build[unknown/unknown/unknown], OS[Linux/5.15.0-130-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7-Ubuntu-122.04.1]
[2025-02-12T16:30:53,273][INFO ][o.o.n.Node               ] JVM home [/usr/lib/jvm/java-21-openjdk-amd64]
[2025-02-12T16:30:53,275][INFO ][o.o.n.Node               ] JVM arguments []
[2025-02-12T16:30:53,281][INFO ][o.o.p.PluginsService     ] no modules loaded
[2025-02-12T16:30:53,282][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.analysis.common.CommonAnalysisPlugin]
[2025-02-12T16:30:53,282][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.transport.Netty4Plugin]
[2025-02-12T16:30:53,307][WARN ][stderr                   ] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-02-12T16:30:53,307][WARN ][stderr                   ] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-02-12T16:30:53,307][WARN ][stderr                   ] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-02-12T16:30:53,342][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2025-02-12T16:30:53,345][INFO ][o.o.e.NodeEnvironment    ] using [1] data paths, mounts [[/ (/dev/md2)]], net usable_space [879.3gb], net total_space [1.9tb], types [ext4]
[2025-02-12T16:30:53,347][INFO ][o.o.e.NodeEnvironment    ] heap size [29.5gb], compressed ordinary object pointers [true]
[2025-02-12T16:30:53,382][WARN ][o.a.l.i.v.VectorizationProvider] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
[2025-02-12T16:30:53,434][INFO ][o.o.n.Node               ] node name [Node 1], node ID [2VRhnHBvTmeFosNgbKsOQw], cluster name [photon], roles [data, cluster_manager]
[2025-02-12T16:30:53,452][INFO ][o.o.e.ExtensionsManager  ] ExtensionsManager initialized
[2025-02-12T16:30:54,125][INFO ][o.o.t.NettyAllocator     ] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2025-02-12T16:30:54,181][INFO ][o.o.d.DiscoveryModule    ] using discovery type [single-node] and seed hosts providers [settings]
[2025-02-12T16:30:54,314][WARN ][o.o.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-02-12T16:30:54,426][INFO ][o.o.n.Node               ] initialized
[2025-02-12T16:30:54,427][INFO ][o.o.n.Node               ] starting ...
[2025-02-12T16:30:54,482][INFO ][o.o.t.TransportService   ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2025-02-12T16:30:54,557][INFO ][o.o.c.c.Coordinator      ] cluster UUID [9l_xaW4JQsCPeqdfqVQ4Ig]
[2025-02-12T16:30:54,617][INFO ][o.o.c.s.MasterService    ] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 4, version: 43, delta: cluster-manager node changed {previous [], current [{Node 1}{2VRhnHBvTmeFosNgbKsOQw}{_GOnUUQRSVOrkrVZ1Huctg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}
[2025-02-12T16:30:54,672][INFO ][o.o.c.s.ClusterApplierService] cluster-manager node changed {previous [], current [{Node 1}{2VRhnHBvTmeFosNgbKsOQw}{_GOnUUQRSVOrkrVZ1Huctg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 43, reason: Publication{term=4, version=43}
[2025-02-12T16:30:54,689][INFO ][o.o.d.PeerFinder         ] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Node 1}{2VRhnHBvTmeFosNgbKsOQw}{_GOnUUQRSVOrkrVZ1Huctg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]
[2025-02-12T16:30:54,691][INFO ][o.o.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[2025-02-12T16:30:54,692][INFO ][o.o.n.Node               ] started
[2025-02-12T16:30:54,692][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:54,706][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:54,729][INFO ][o.o.g.GatewayService     ] recovered [1] indices into cluster_state
[2025-02-12T16:30:54,762][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:54,814][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:54,933][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,140][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,216][INFO ][o.o.c.r.a.AllocationService] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[photon][4]]]).
[2025-02-12T16:30:55,232][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,389][INFO ][d.k.p.App                ] Make sure that the ES cluster is ready, this might take some time.
[2025-02-12T16:30:55,478][INFO ][d.k.p.App                ] ES cluster is now ready.
[2025-02-12T16:30:55,546][INFO ][o.o.c.m.MetadataIndexStateService] closing indices [photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:55,590][INFO ][o.o.c.m.MetadataIndexStateService] completed closing of indices [photon]
[2025-02-12T16:30:55,591][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,640][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,659][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:55,688][INFO ][o.o.c.m.MetadataIndexStateService] opening indices [[photon/55FGuBZoSQGwS-skSIpZ9Q]]
[2025-02-12T16:30:55,691][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:55,706][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,723][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:55,807][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,853][INFO ][o.o.c.r.a.AllocationService] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[photon][3]]]).
[2025-02-12T16:30:55,866][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-02-12T16:30:55,886][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/55FGuBZoSQGwS-skSIpZ9Q]
[2025-02-12T16:30:55,943][INFO ][o.e.j.u.log              ] Logging initialized @3139ms to org.eclipse.jetty.util.log.Slf4jLog
[2025-02-12T16:30:55,969][INFO ][s.e.j.EmbeddedJettyServer] == Spark has ignited ...
[2025-02-12T16:30:55,969][INFO ][s.e.j.EmbeddedJettyServer] >> Listening on 0.0.0.0:2322
[2025-02-12T16:30:55,971][INFO ][o.e.j.s.Server           ] jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 21.0.6+7-Ubuntu-122.04.1
[2025-02-12T16:30:55,981][INFO ][o.e.j.s.session          ] DefaultSessionIdManager workerName=node0
[2025-02-12T16:30:55,981][INFO ][o.e.j.s.session          ] No SessionScavenger set, using defaults
[2025-02-12T16:30:55,982][INFO ][o.e.j.s.session          ] node0 Scavenging every 660000ms
[2025-02-12T16:30:55,987][INFO ][o.e.j.s.AbstractConnector] Started ServerConnector@25fe4353{HTTP/1.1, (http/1.1)}{0.0.0.0:2322}
[2025-02-12T16:30:55,987][INFO ][o.e.j.s.Server           ] Started @3184ms

BUT:

curl -v "http://127.0.0.1:2322/structured?q=berlin"
*   Trying 127.0.0.1:2322...
* Connected to 127.0.0.1 (127.0.0.1) port 2322 (#0)
> GET /structured?q=berlin HTTP/1.1
> Host: 127.0.0.1:2322
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Wed, 12 Feb 2025 16:32:25 GMT
< Content-Type: application/json;charset=utf-8
< Transfer-Encoding: chunked
< Server: Jetty(9.4.56.v20240826)
< 
* Connection #0 to host 127.0.0.1 left intact

photon logfile of this request:

[2025-02-12T16:32:25,767][INFO ][s.h.m.MatcherFilter      ] The requested route [/structured] has not been mapped in Spark for Accept: [*/*]

What am I missing for getting structured query goodness as described here: https://github.com/komoot/photon/blob/master/docs/structured.md ?

[lonvia]

The pre-computed databases do not have the indexes for structured search enabled. You need to build your own index from a Nominatim database.

We probably could also add functions to create the necessary indexes after the fact. I've added #868 to that effect. Help welcome.

[lonvia]

For more information on structured search, see https://github.com/komoot/photon/blob/master/docs/structured.md. In particular, note that you need to run the import from Nominatim with the parameter '-structured'.