Connection issues during upstream transition to AAACertificateServices #872

fzyzcjy · 2021-03-31T06:44:39Z

fzyzcjy
Mar 31, 2021

Thank you for the wonderful library!

I am using jib (with distroless as base image) to build a Spring Boot app with pushy library connecting to Apple APNs.

I have the problem about certificates in 2021.3.31. Months ago there was this problem and it was solved, but today it does not work. I suspect it is related to AAACertificate, but after a change, it still does not work

related: #833 (comment)

Details of what I have done:

build the client using:

        File trustedServerCertificatePemFile = ResourceUtils
                .getFile(config.getTrustedServerCertificatePemClasspathFile());

        return new ApnsClientBuilder()
                .setApnsServer(...)
                .setSigningKey(...)
                .setMetricsListener(...)
                .setTrustedServerCertificateChain(trustedServerCertificatePemFile)
                .build();

where the path is classpath:AAACertificateServices.crt

put a file named AAACertificateServices.crt under /resources folder with content:

error:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1793)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:777)
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511)
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1086)
	at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:977)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1450)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
	... 17 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
	at sun.security.validator.Validator.validate(Validator.java:271)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261)
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:698)
	at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:596)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1220)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1346)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1389)
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387)
	... 21 common frames omitted
	Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
		at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1309)
		at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1270)
		... 25 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)
	... 35 common frames omitted

Answered by jchambers

Mar 31, 2021

Folks, this sounds like a request for assistance more than a request for an engineering change. I'm going to convert this into a discussion rather than an issue.

That said, it appears that Apple is sometimes sending the old GeoTrust certificate and sometimes sending the new AAACertificateServices certificate. You can check which you're getting with:

openssl s_client -connect api.push.apple.com:443 -showcerts -servername api.push.apple.com

I'm guessing that this is a case where the upstream change has been only partially rolled out, and there are a mix of old and new certificates in play.

@fzyzcjy the difficulty is that you're setting exactly one trusted certificate (I presume the AAA cert…

View full answer

lorecioni · 2021-03-31T07:57:39Z

lorecioni
Mar 31, 2021

@fzyzcjy @LightySnake we still faced up the problem too, also mentioned in #870

We tried to add the full setTrustedServerCertificateChain using the Apple newly released certificates in https://developer.apple.com/news/?id=7gx0a2lp.

CertificateFactory certFactory;
    certFactory = CertificateFactory.getInstance("X.509");
    X509Certificate cert1 = (X509Certificate) certFactory.generateCertificate(ca1Resource.getInputStream());
    X509Certificate cert2 = (X509Certificate) certFactory.generateCertificate(ca2Resource.getInputStream());
    X509Certificate cert3 = (X509Certificate) certFactory.generateCertificate(ca3Resource.getInputStream());

    ApnsClient apnsClient = new ApnsClientBuilder()
            .setApnsServer(ApnsClientBuilder.PRODUCTION_APNS_HOST)
            .setClientCredentials(certificate, certPassword)
            .setTrustedServerCertificateChain(cert1, cert2, cert3)
            .build();

0 replies

fzyzcjy · 2021-03-31T08:07:44Z

fzyzcjy
Mar 31, 2021
Author

@jchambers Dear maintainer, could you please provide some suggestions? Thank you very much!

0 replies

jchambers · 2021-03-31T14:51:36Z

jchambers
Mar 31, 2021
Maintainer

Folks, this sounds like a request for assistance more than a request for an engineering change. I'm going to convert this into a discussion rather than an issue.

That said, it appears that Apple is sometimes sending the old GeoTrust certificate and sometimes sending the new AAACertificateServices certificate. You can check which you're getting with:

openssl s_client -connect api.push.apple.com:443 -showcerts -servername api.push.apple.com

I'm guessing that this is a case where the upstream change has been only partially rolled out, and there are a mix of old and new certificates in play.

@fzyzcjy the difficulty is that you're setting exactly one trusted certificate (I presume the AAA certificate), and so the connection fails if the server presents the old GeoTrust certificate. As Apple called out in their announcement about this change:

To ensure a seamless transition and to avoid push notification delivery failures, verify that both the old and new root certificates for the HTTP/2 interface are included in the Trust Store of each of your notification servers before March 29.

I had hoped that they meant that this was necessary because there would be a smooth, one-way transition from the old certificate to the new, but it appears that we may be dealing with an extended period in which both certificates are in play.

4 replies

fzyzcjy Apr 1, 2021
Author

@jchambers Excellent! Now it works after putting two certificates! Thank you very much!

LightySnake Apr 1, 2021

@jchambers Excellent! Now it works after putting two certificates! Thank you very much!

Hi @fzyzcjy can you give me the list of the specific certificates that you used? You add the certificates to the app using setTrustedServerCertificateChain or you add it also to the server?

Thank you

fzyzcjy Apr 1, 2021
Author

@LightySnake

Code: (you may not have some helper functions, but the major idea is quite simple)

    @SneakyThrows
    private X509Certificate[] getTrustedServerCertificates(ExternalPushIosConfig config) {
        List<File> files = config.getTrustedServerCertificatePemClasspathFiles().stream()
                .map(U.catchThrowableFunction(ResourceUtils::getFile))
                .peek(file -> checkArgument(file.exists(),
                        "The trustedServerCertificatePemFile does not exist on disk: " + file.getAbsolutePath()))
                .collect(Collectors.toList());

        CertificateFactory certFactory = CertificateFactory.getInstance("X.509");

        return files.stream().map(U.catchThrowableFunction(file -> {
            try (InputStream is = FileUtils.openInputStream(file)) {
                return (X509Certificate) certFactory.generateCertificate(is);
            }
        })).toArray(X509Certificate[]::new);
    }

void yourmethod() {
        X509Certificate[] trustedServerCertificates = getTrustedServerCertificates(config);
        return new ApnsClientBuilder()
...
                .setTrustedServerCertificateChain(trustedServerCertificates)
...
                .build();
   
}

AAACertificateServices.crt

GeoTrust_Global_CA.pem

LightySnake Apr 1, 2021

Thank you! I will try

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection issues during upstream transition to AAACertificateServices #872

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Connection issues during upstream transition to AAACertificateServices #872

fzyzcjy Mar 31, 2021

Replies: 3 comments · 4 replies

lorecioni Mar 31, 2021

fzyzcjy Mar 31, 2021 Author

jchambers Mar 31, 2021 Maintainer

fzyzcjy Apr 1, 2021 Author

LightySnake Apr 1, 2021

fzyzcjy Apr 1, 2021 Author

LightySnake Apr 1, 2021

fzyzcjy
Mar 31, 2021

Replies: 3 comments 4 replies

lorecioni
Mar 31, 2021

fzyzcjy
Mar 31, 2021
Author

jchambers
Mar 31, 2021
Maintainer

fzyzcjy Apr 1, 2021
Author

fzyzcjy Apr 1, 2021
Author