Investigate recurrent indexing failures

[yrodiere]

See #130 (comment)

It's happening regulary, there must be something wrong.

I know @marko-bekhta mentioned something about OpenSearch restarting. Maybe something is wrong with its config, and it's crashing?

Stacktrace:

java.lang.IllegalStateException: Failed to create indexes: HSEARCH000520: Hibernate Search encountered failures during Schema management. Failures:

    type 'io.quarkus.search.app.entity.Guide': 
        failures: 
          - HSEARCH400034: Unable to retrieve index metadata from Elasticsearch: HSEARCH400007: Elasticsearch request failed: search-backend-0.search-backend: Name or service not known
            Request: GET /guide-write,guide-read with parameters {ignore_unavailable=true, allow_no_indices=true}
            Response: (no response)
	at io.quarkus.search.app.indexing.IndexingService.createIndexesIfMissing(IndexingService.java:228)
	at io.quarkus.search.app.indexing.IndexingService.reindex(IndexingService.java:193)
	at io.quarkus.search.app.indexing.IndexingService_Subclass.reindex$$superforward(Unknown Source)
	at io.quarkus.search.app.indexing.IndexingService_Subclass$$function$$1.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.arc.impl.ActivateRequestContextInterceptor.invoke(ActivateRequestContextInterceptor.java:124)
	at io.quarkus.arc.impl.ActivateRequestContextInterceptor.aroundInvoke(ActivateRequestContextInterceptor.java:33)
	at io.quarkus.arc.impl.ActivateRequestContextInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at io.quarkus.search.app.indexing.IndexingService_Subclass.reindex(Unknown Source)
	at io.quarkus.search.app.indexing.IndexingService.lambda$scheduleIndexing$9(IndexingService.java:179)
	at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:640)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1495)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during Schema management. Failures:

    type 'io.quarkus.search.app.entity.Guide': 
        failures: 
          - HSEARCH400034: Unable to retrieve index metadata from Elasticsearch: HSEARCH400007: Elasticsearch request failed: search-backend-0.search-backend: Name or service not known
            Request: GET /guide-write,guide-read with parameters {ignore_unavailable=true, allow_no_indices=true}
            Response: (no response)
	at org.hibernate.search.engine.reporting.spi.RootFailureCollector.checkNoFailure(RootFailureCollector.java:65)
	at org.hibernate.search.mapper.pojo.standalone.schema.management.impl.SearchSchemaManagerImpl.doOperation(SearchSchemaManagerImpl.java:81)
	at org.hibernate.search.mapper.pojo.standalone.schema.management.impl.SearchSchemaManagerImpl.createIfMissing(SearchSchemaManagerImpl.java:36)
	at io.quarkus.search.app.indexing.IndexingService.createIndexesIfMissing(IndexingService.java:206)
	... 25 more
	Suppressed: org.hibernate.search.util.common.SearchException: HSEARCH400034: Unable to retrieve index metadata from Elasticsearch: HSEARCH400007: Elasticsearch request failed: search-backend-0.search-backend: Name or service not known
Request: GET /guide-write,guide-read with parameters {ignore_unavailable=true, allow_no_indices=true}
Response: (no response)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaAccessor.lambda$getCurrentIndexMetadata$0(ElasticsearchSchemaAccessor.java:96)
		at org.hibernate.search.util.common.impl.Futures.lambda$handler$1(Futures.java:60)
		at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
		at java.base/java.util.concurrent.CompletableFuture.uniExceptionallyStage(CompletableFuture.java:1008)
		at java.base/java.util.concurrent.CompletableFuture.exceptionally(CompletableFuture.java:2364)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaAccessor.getCurrentIndexMetadata(ElasticsearchSchemaAccessor.java:95)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaAccessor.getCurrentIndexMetadataOrNull(ElasticsearchSchemaAccessor.java:85)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaCreator.createIndexIfAbsent(ElasticsearchSchemaCreator.java:64)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchIndexSchemaManager.createIfMissing(ElasticsearchIndexSchemaManager.java:62)
		at org.hibernate.search.mapper.pojo.schema.management.impl.PojoScopeSchemaManagerImpl.lambda$doOperationOnTypesBiFunction$0(PojoScopeSchemaManagerImpl.java:87)
		at org.hibernate.search.mapper.pojo.schema.management.impl.PojoScopeSchemaManagerImpl.doOperationOnTypesTriFunction(PojoScopeSchemaManagerImpl.java:104)
		at org.hibernate.search.mapper.pojo.schema.management.impl.PojoScopeSchemaManagerImpl.doOperationOnTypesBiFunction(PojoScopeSchemaManagerImpl.java:86)
		at org.hibernate.search.mapper.pojo.schema.management.impl.PojoScopeSchemaManagerImpl.createIfMissing(PojoScopeSchemaManagerImpl.java:36)
		at org.hibernate.search.mapper.pojo.standalone.schema.management.impl.SearchSchemaManagerImpl.doOperation(SearchSchemaManagerImpl.java:75)
		... 27 more
	Caused by: org.hibernate.search.util.common.SearchException: HSEARCH400007: Elasticsearch request failed: search-backend-0.search-backend: Name or service not known
Request: GET /guide-write,guide-read with parameters {ignore_unavailable=true, allow_no_indices=true}
Response: (no response)
		at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.lambda$execute$2(AbstractNonBulkableWork.java:62)
		at org.hibernate.search.util.common.impl.Futures.lambda$handler$1(Futures.java:60)
		at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
		at java.base/java.util.concurrent.CompletableFuture.uniExceptionallyStage(CompletableFuture.java:1008)
		at java.base/java.util.concurrent.CompletableFuture.exceptionally(CompletableFuture.java:2364)
		at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.execute(AbstractNonBulkableWork.java:54)
		at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchSimpleWorkOrchestrator$WorkExecution.execute(ElasticsearchSimpleWorkOrchestrator.java:66)
		at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchSimpleWorkOrchestrator.doSubmit(ElasticsearchSimpleWorkOrchestrator.java:40)
		at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchSimpleWorkOrchestrator.doSubmit(ElasticsearchSimpleWorkOrchestrator.java:15)
		at org.hibernate.search.engine.backend.orchestration.spi.AbstractWorkOrchestrator.submit(AbstractWorkOrchestrator.java:137)
		at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchSimpleWorkOrchestrator.submit(ElasticsearchSimpleWorkOrchestrator.java:28)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaAccessor.execute(ElasticsearchSchemaAccessor.java:194)
		at org.hibernate.search.backend.elasticsearch.schema.management.impl.ElasticsearchSchemaAccessor.getCurrentIndexMetadata(ElasticsearchSchemaAccessor.java:94)
		... 35 more
	Caused by: java.net.UnknownHostException: search-backend-0.search-backend: Name or service not known
		at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
		at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
		at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
		at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
		at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
		at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:666)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:637)
		at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:472)
		at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
		at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:381)
		at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:130)
		at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:116)
		at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138)
		at org.elasticsearch.client.RestClient.lambda$performRequestAsync$0(RestClient.java:397)
		at org.elasticsearch.client.Cancellable$RequestCancellable.runIfNotCancelled(Cancellable.java:90)
		at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:395)
		at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:382)
		at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl.send(ElasticsearchClientImpl.java:112)
		at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl.lambda$submit$0(ElasticsearchClientImpl.java:82)
		at org.hibernate.search.util.common.impl.Futures.lambda$create$0(Futures.java:43)
		at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
		at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309)
		at org.hibernate.search.util.common.impl.Futures.create(Futures.java:43)
		at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl.submit(ElasticsearchClientImpl.java:82)
		at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.lambda$execute$1(AbstractNonBulkableWork.java:53)
		at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
		at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309)
		at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.execute(AbstractNonBulkableWork.java:53)
		... 42 more
	Suppressed: java.lang.IllegalStateException: Failed to recover aliases: search-backend-0.search-backend: Name or service not known
		at io.quarkus.search.app.indexing.Rollover.recoverInconsistentAliases(Rollover.java:100)
		at io.quarkus.search.app.indexing.IndexingService.createIndexesIfMissing(IndexingService.java:220)
		... 25 more
	Caused by: java.io.IOException: search-backend-0.search-backend: Name or service not known
		at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:939)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
		at io.quarkus.search.app.indexing.Rollover.aliased(Rollover.java:175)
		at io.quarkus.search.app.indexing.Rollover.recoverInconsistentAliases(Rollover.java:95)
		... 26 more
	Caused by: java.net.UnknownHostException: search-backend-0.search-backend: Name or service not known
		at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
		at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
		at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
		at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
		at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
		at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:666)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:637)
		at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:472)
		at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
		at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
		at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:381)
		at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:130)
		at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:116)
		at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
		... 29 more