Light-4J (2.0.20) - wrong IPs is used in the LightCluster.discovery() method

[mihai-vladuc]

Here’s a summary of the issue as per chronological order.

1, When Light-4J starts, the serviceMap container has the target IP (e.g. 10.10.10.10)

2. When target changes IP (e.g. 11.11.11.11) , this method will first look at the cache (serviceMap) and if there exists use that.

3. Issue is the serviceMap contains old IP (10.10.10.10)

4. Hence wrong IP being used and hence it blows up since IP 10.10.10.10 no longer exists

Here’s the code:

private List discovery(String protocol, String serviceId, String tag) {
if(logger.isDebugEnabled()) logger.debug("protocol = " + protocol + " serviceId = " + serviceId);
// lookup in serviceMap first, if not there, then subscribe and discover.
List urls = serviceMap.get(serviceId);
if(logger.isDebugEnabled()) logger.debug("cached serviceId " + serviceId + " urls = " + urls);
if((urls == null) || (urls.isEmpty())) {
URL subscribeUrl = URLImpl.valueOf(protocol + "://localhost/" + serviceId);
if(tag != null) {
subscribeUrl.addParameter(Constants.TAG_ENVIRONMENT, tag);

[stevehu]

There are several cases the IP might be changed and I am not sure which one is your case and what is your environment. In all cases, we need to restart the target service so that the newly updated IP will be registered to the Consul or light-portal. May I ask why the IP is changed on the target server and the server still running? Thanks.

[mihai-vladuc]

We'll add more information once it becomes available, meanwhile, we have asked the person who open this issue to upgrade to the latest light-4j 2.1.37.

[mihai-vladuc]

@stevehu , here is more updates from our client :

The version 2.1.27 has the same bug reported (#2359 (2.1.21 to 2.1.37) possible bugs on LightCluster & AbstractRegistry classes).
The fix Steve Hu did for bug (#2359) was merged into master branch which is 2.1.38-SNAPSHOT. I use the master branch and it produces a different error.
The error stemmed somewhere in javax.net.ssl.SSLContextSpi.getDefaultSocket() as outlined below.

ERROR com.networknt.handler.Handler:426 initStringDefinedHandler - Could not instantiate handler class class com.#######.cp.router.handler.DataPostHandler.  
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.networknt.handler.Handler.initStringDefinedHandler(Handler.java:423)
at com.networknt.handler.Handler.initHandlers(Handler.java:82)
at com.networknt.handler.Handler.init(Handler.java:64)
at com.networknt.server.Server.start(Server.java:181)
at com.networknt.server.Server.init(Server.java:132)
at com.networknt.server.Server.main(Server.java:105)
Caused by: java.lang.IllegalStateException: SSLContext is not initialized <==== root cause
at java.base/sun.security.ssl.SSLContextImpl.engineGetSocketFactory(SSLContextImpl.java:208)
at java.base/javax.net.ssl.SSLContextSpi.getDefaultSocket(SSLContextSpi.java:162)
at java.base/javax.net.ssl.SSLContextSpi.engineGetSupportedSSLParameters(SSLContextSpi.java:212)
at java.base/javax.net.ssl.SSLContext.getSupportedSSLParameters(SSLContext.java:471)
at java.net.http/jdk.internal.net.http.HttpClientImpl.getDefaultParams(HttpClientImpl.java:334)
at java.net.http/jdk.internal.net.http.HttpClientImpl.<init>(HttpClientImpl.java:299)

I haven't had the time to relook since i don't have the source code for javax.net.ssl.SSLContextSpi but it failed there.
Let me know if you need anything new.
Thanks.

[stevehu]

From the stack trace, it looks like the problem is in one of your handlers. The class name is "com.#######.cp.router.handler.DataPostHandler". Please check the code in this handler to see if it opens a connection to another service with HTTPS? If yes, please double-check the certificates. Thanks.

[mihai-vladuc]

@stevehu : Thank You! I'll forward your comment.

[mihai-vladuc]

From our client:
I have tested the fix Steve Hu did on the master branch (2.1.38-SNAPSHOT) and it’s not working. Here’s a summary.

The consul URI managed to be extracted (black arrow in the screen shot)
The serviceToUrl() is return null even though tag “Dev” is passed onto its parameter.
Summary on ConsulRegistry class.

In the run() method, this seems to always populate just the service name once the run() kicks off.
updateServiceCache(serviceName, serviceUrls, true); This will populate serviceCache key with no "|Dev" since serviceName is without tag.

2. In the updateServiceCache(), the very first time it ran:

if (serviceUrls != null && !serviceUrls.isEmpty()) {
List cachedUrls = serviceCache.get(key); // null due to serviceCache key has no tag "|Dev"
List newUrls = serviceUrls.get(key); // null

boolean change = true;
if (ConsulUtils.isSame(newUrls, cachedUrls)) { // This will always be true.
change = false;
} else {
serviceCache.put(key, newUrls); // this will not get invoked
}
This is urgent and requires an a fix ASAP.

Please advice.

[yapkm01]

Hi Steve .. In a nutshell the service discovery for Light-4J version 2.1.38-SNAPSHOT is not working when there's a tag passed in to serviceToUrl method. Tx

[stevehu]

Are you migrating from the 1.6.x? Is your app working with 1.6.x? If yes, I will try to port the logic from 1.6.x to 2.1.x. Thanks.