Pubby deploy hangs forever in Tomcat 8

[seweissman]

I am using Pubby with OpenRDF Sesame running on the same Tomcat server. The problem appears to be that if the Sesame repository is not already loaded when Pubby starts, Pubby hangs forever, which means that the Tomcat manager never starts and the only way to get things working again is to manually remove Pubby from the Tomcat webapp directory.

It would be great if Pubby could just fail and not hang up things forever if it can't find the back-end server. (Maybe setting a connect timeout on a URLConnection somewhere would fix this?)

[cygri]

Sarah, may I ask for some more detail? So you're running Sesame and Pubby in the same Tomcat? Did you get Pubby working against a different backend? What makes you believe that the issue is that Sesame is not yet loaded? In what way does it “start working again” after deleting the Pubby webapp dir? Thanks!

(Usually we want Pubby to fail as soon as possible if there's any problem, so that users get fast feedback. That may be a problem if it relies on a backend that is starting up in parallel...)

[seweissman]

Pubby works with Sesame when I load the Sesame server first then load the Pubby war. It is just when Tomcat restarts and tries to reload everything (including Sesame and pubby) that there is a problem with hanging. By "start working again", I just mean that in order to get Tomcat to finish its loading process, I have to delete Pubby manually from the directory where Tomcat stores its webapps, Then I can redeploy Pubby through the manager GUI for Tomcat like I would deploy most webapps. But this means I'd have to undeploy Pubby every time I want to restart the server. (And sometimes sysadmins go ahead and restart things without telling me.)

I did a thread dump on the hanging process and this is one of the threads that is running, and is what I think is holding things up. You can see that it is RemoteSPARQLDataSource.execQuerySelect that is causing this socket read:

"localhost-startStop-1" daemon prio=10 tid=0x0000000000f4b800 nid=0x451c runnable    [0x00007fb2e1588000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:137)
at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:118)
at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1011)
at org.apache.jena.riot.web.HttpOp.execHttpGet(HttpOp.java:291)
at org.apache.jena.riot.web.HttpOp.execHttpGet(HttpOp.java:353)
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:326)
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:276)
at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:346)
at de.fuberlin.wiwiss.pubby.sources.RemoteSPARQLDataSource.execQuerySelect(RemoteSPARQLDataSource.java:299)
at de.fuberlin.wiwiss.pubby.sources.RemoteSPARQLDataSource.getIndex(RemoteSPARQLDataSource.java:212)
at de.fuberlin.wiwiss.pubby.sources.RewrittenDataSource.getIndex(RewrittenDataSource.java:129)
at de.fuberlin.wiwiss.pubby.sources.FilteredDataSource.getIndex(FilteredDataSource.java:64)
at de.fuberlin.wiwiss.pubby.sources.MergeDataSource.getIndex(MergeDataSource.java:104)
at de.fuberlin.wiwiss.pubby.Configuration.<init>(Configuration.java:122)
at de.fuberlin.wiwiss.pubby.Configuration.create(Configuration.java:41)
at de.fuberlin.wiwiss.pubby.servlets.ServletContextInitializer.contextInitialized(ServletContextInitializer.java:37)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4797)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5221)
- locked <0x00000007c7e31138> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0x00000007c7e31138> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

[seweissman]

If I add a timeout before the call to execSelect, this seems to take care of the problem, although I could not say what the right timeout should be.

private ResultSet execQuerySelect(String query) {
    QueryEngineHTTP endpoint = new QueryEngineHTTP(endpointURL, query);
    if (defaultGraphURI != null) {
        endpoint.setDefaultGraphURIs(Collections.singletonList(defaultGraphURI));
    }
    for (String[] param: queryParamsSelect) {
        endpoint.addParam(param[0], param[1]);
    }
            endpoint.setTimeout(10000, 10000);
    return endpoint.execSelect();
}

[cygri]

Thanks for investigating this, Sarah. Seems like you have at least a temporary workaround.

Pubby executes a query during startup to populate the index resource. That is the hanging query. We may need a switch that changes Pubby's startup behaviour. If it is started as a stand-alone app, it should start quickly and fail quickly. If it is started as part of a see let container, it should defer this operation, at least until the actual servlet is started, and perhaps even until the first client request that needs it.