Dependency conflicts on commons-codec:commons-codec, leading to inconsistent program behaviors

[HelloCoCooo]

Hi, in weixin4j-master/weixin4j-base, there are mulptiple versions of library commons-codec:commons-codec. However, according to Maven's dependency management strategy: "first declaration wins", only commons-codec:commons-codec:1.2 can be loaded, and commons-codec:commons-codec:1.6 will be shadowed.

As shown in the following figure, your project expects to invoke method <org.apache.commons.codec.binary.Base64: decodeBase64([B)[B> in library commons-codec:commons-codec:1.6 (along the original dependency path). As it has been shadowed, this method defined in commons-codec:commons-codec:1.2 is actually forced to be referenced via the following invocation path (along the actual dependency path):

<com.foxinmy.weixin4j.http.support.apache4.HttpComponent4_2: execute(Lcom/foxinmy/weixin4j/http/HttpRequest;)Lcom/foxinmy/weixin4j/http/HttpResponse;> /home/wwww/wangSensor/unzip/weixin4j-master/weixin4j-base/target/classes
<org.apache.http.impl.client.CloseableHttpClient: execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.client.CloseableHttpClient: execute(Lorg/apache/http/client/methods/HttpUriRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse;> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.client.InternalHttpClient: doExecute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse;> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.execchain.MainClientExec: execute(Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/client/methods/HttpRequestWrapper;Lorg/apache/http/client/protocol/HttpClientContext;Lorg/apache/http/client/methods/HttpExecutionAware;)Lorg/apache/http/client/methods/CloseableHttpResponse;> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.execchain.MainClientExec: needAuthentication(Lorg/apache/http/auth/AuthState;Lorg/apache/http/auth/AuthState;Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/HttpResponse;Lorg/apache/http/client/protocol/HttpClientContext;)Z> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.auth.HttpAuthenticator: handleAuthChallenge(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpResponse;Lorg/apache/http/client/AuthenticationStrategy;Lorg/apache/http/auth/AuthState;Lorg/apache/http/protocol/HttpContext;)Z> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.auth.AuthSchemeBase: processChallenge(Lorg/apache/http/Header;)V> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.http.impl.auth.GGSSchemeBase: parseChallenge(Lorg/apache/http/util/CharArrayBuffer;II)V> /home/wwww/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
<org.apache.commons.codec.binary.Base64: decodeBase64([B)[B>

Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----

Code snippet of <org.apache.commons.codec.binary.Base64: decodeBase64([B)[B> in commons-codec:commons-codec:1.6 (shadowed but expected to invoke method):

detailed method body

public static byte[] decodeBase64(byte[] base64Data) {
        return new Base64().decode(base64Data);
    }

public byte[] decode(byte[] pArray) {
        reset();
        if (pArray == null || pArray.length == 0) {
            return pArray;
        }
        decode(pArray, 0, pArray.length);
        decode(pArray, 0, -1); // Notify decoder of EOF.
        byte[] result = new byte[pos];
        readResults(result, 0, result.length);
        return result;
    }

Code snippet of <org.apache.commons.codec.binary.Base64: decodeBase64([B)[B> in commons-codec:commons-codec:1.2 (loaded version):

detailed method body

public static byte[] decodeBase64(byte[] base64Data) {
        base64Data = discardNonBase64(base64Data);
        if (base64Data.length == 0) {
            return new byte[0];
        } else {
            int numberQuadruple = base64Data.length / 4;
            byte[] decodedData = null;
            byte b1 = false;
            byte b2 = false;
            byte b3 = false;
            byte b4 = false;
            byte marker0 = false;
            byte marker1 = false;
            int encodedIndex = 0;
            int dataIndex = false;
            int i = base64Data.length;

            while(base64Data[i - 1] == 61) {
                --i;
                if (i == 0) {
                    return new byte[0];
                }
            }

            byte[] decodedData = new byte[i - numberQuadruple];

            for(i = 0; i < numberQuadruple; ++i) {
                int dataIndex = i * 4;
                byte marker0 = base64Data[dataIndex + 2];
                byte marker1 = base64Data[dataIndex + 3];
                byte b1 = base64Alphabet[base64Data[dataIndex]];
                byte b2 = base64Alphabet[base64Data[dataIndex + 1]];
                byte b3;
                if (marker0 != 61 && marker1 != 61) {
                    b3 = base64Alphabet[marker0];
                    byte b4 = base64Alphabet[marker1];
                    decodedData[encodedIndex] = (byte)(b1 << 2 | b2 >> 4);
                    decodedData[encodedIndex + 1] = (byte)((b2 & 15) << 4 | b3 >> 2 & 15);
                    decodedData[encodedIndex + 2] = (byte)(b3 << 6 | b4);
                } else if (marker0 == 61) {
                    decodedData[encodedIndex] = (byte)(b1 << 2 | b2 >> 4);
                } else if (marker1 == 61) {
                    b3 = base64Alphabet[marker0];
                    decodedData[encodedIndex] = (byte)(b1 << 2 | b2 >> 4);
                    decodedData[encodedIndex + 1] = (byte)((b2 & 15) << 4 | b3 >> 2 & 15);
                }

                encodedIndex += 3;
            }

            return decodedData;
        }
    }

Dependency tree--

[INFO] com.foxinmy:weixin4j-base:jar:1.9.0-SNAPSHOT
[INFO] +- com.alibaba:fastjson:jar:1.2.31:compile
[INFO] +- junit:junit:jar:4.8.2:test
[INFO] +- commons-httpclient:commons-httpclient:jar:3.0:compile
[INFO] | +- (junit:junit:jar:4.8.2:test - version managed from 3.8.1; scope managed from compile; omitted for duplicate)
[INFO] | +- commons-logging:commons-logging:jar:1.0.3:compile
[INFO] | - commons-codec:commons-codec:jar:1.2:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
[INFO] | +- (commons-logging:commons-logging:jar:1.1.3:compile - omitted for conflict with 1.0.3)
[INFO] | - (commons-codec:commons-codec:jar:1.6:compile - omitted for conflict with 1.2)
[INFO] +- io.netty:netty-all:jar:4.1.42.Final:compile
[INFO] +- com.squareup.okhttp3:okhttp:jar:3.4.1:compile
[INFO] | - com.squareup.okio:okio:jar:1.9.0:compile
[INFO] +- com.squareup.okhttp:okhttp:jar:2.7.5:compile
[INFO] | - (com.squareup.okio:okio:jar:1.6.0:compile - omitted for conflict with 1.9.0)
[INFO] +- redis.clients:jedis:jar:2.8.1:compile
[INFO] | - org.apache.commons:commons-pool2:jar:2.4.2:compile
[INFO] +- com.whalin:Memcached-Java-Client:jar:3.0.2:compile
[INFO] | +- commons-pool:commons-pool:jar:1.5.6:compile
[INFO] | - org.slf4j:slf4j-api:jar:1.6.4:compile
[INFO] +- org.bouncycastle:bcprov-jdk16:jar:1.46:compile
[INFO] - javax.xml.bind:jaxb-api:jar:2.2.11:provided

Suggested solutions:

Solution1: Declare version commons-codec:commons-codec:1.6 as a direct dependency, to override the version 1.2 (based on Maven's nearest wins loading strategy).

Solution2: reversing the declaration order of these two libraries in pom file.

Thanks.
Best regards,
Coco

[HelloCoCooo]

Executing the following test case on commons-codec:commons-codec:1.2 and 1.6 separately, the risky method <org.apache.commons.codec.binary.Base64: decodeBase64([B)[B> will get different return values:

@Test(timeout = 4000)
    public void test01()  throws Throwable  {
        Base64 base64 = new Base64();
        byte[] byteArray0 = new byte[]{(byte) 25, (byte) 76, (byte) 56};
        byte[] byteArray1 = base64.decodeBase64(byteArray0);
        assertEquals(1, byteArray1.length);
    }

Output results:

byteArray1.length == 1   //On **commons-codec:commons-codec:1.6**

byteArray1.length == 2    //On **commons-codec:commons-codec:1.2**

Variable token (defined in class org.apache.http.impl.auth.GGSSchemeBase of library org.apache.httpcomponents:httpclient:jar:4.3.6) is assigned by the return value of method <org.apache.commons.codec.binary.Base64: decodeBase64([B)[B>.

As such, token value would be changed when the client project references commons-codec:commons-codec:1.2 (compared with the shadowed but expected version 1.6), which could affect program semantic behaviors.

@Override
    protected void parseChallenge(
            final CharArrayBuffer buffer,
            final int beginIndex, final int endIndex) throws MalformedChallengeException {
        final String challenge = buffer.substringTrimmed(beginIndex, endIndex);
        if (log.isDebugEnabled()) {
            log.debug("Received challenge '" + challenge + "' from the auth server");
        }
        if (state == State.UNINITIATED) {
            token = Base64.decodeBase64(challenge.getBytes());
            state = State.CHALLENGE_RECEIVED;
        } else {
            log.debug("Authentication already attempted");
            state = State.FAILED;
        }
    }

[HelloCoCooo]

@foxinmy Could please help me check this issue?
May I pull a request to fix it?
Thanks again.

[foxinmy]

the httpclient dependency is optional，you can declare version commons-codec:commons-codec:1.6 in your pom.xml file

[HelloCoCooo]

@foxinmy Thank you very much for your kindly feedback.
The scope configuariton of dependency commons-httpclient:commons-httpclient:jar:3.0 is "compile".
Could you please revise this issue in your pom.xml, which will benefit many weixin4j's downstream users.
I can also submit a PR to solve it.

Thanks again.

[foxinmy]

虽然scope是compile，但是optional在打包时会排除掉而且不是产生传递依赖，所以不必担心

另外 我能看看你项目里面的pom.xml文件吗？