Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Json Unmarshal Exception with AWS java SDK 2.X #108

Open
KKcorps opened this issue Jan 21, 2021 · 3 comments
Open

Json Unmarshal Exception with AWS java SDK 2.X #108

KKcorps opened this issue Jan 21, 2021 · 3 comments

Comments

@KKcorps
Copy link

KKcorps commented Jan 21, 2021

I am using AWS java SDK 2.15.X and it is throwing the following exception while reading the records from kinesalite

software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (Unable to parse date : 1.611248009879E9). Response Code: 200, Response Text: OK

All the other requests such as Create Stream, put records, describe stream etc. work fine.
@KKcorps
Copy link
Author

KKcorps commented Jan 21, 2021 •
edited
Loading

Complete stackTrace

	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[aws-core-2.15.67.jar:?]
	at software.amazon.awssdk.services.kinesis.DefaultKinesisClient.getRecords(DefaultKinesisClient.java:1051) ~[kinesis-2.15.67.jar:?]
	at org.apache.pinot.plugin.stream.kinesis.KinesisConsumer.getResult(KinesisConsumer.java:99) ~[classes/:?]
	at org.apache.pinot.plugin.stream.kinesis.KinesisConsumer.lambda$fetchMessages$0(KinesisConsumer.java:65) ~[classes/:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_265]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to parse date : 1.611248009879E9
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.lambda$safeParseDate$0(StringToInstant.java:77) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.convert(StringToInstant.java:56) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.convert(StringToInstant.java:32) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$SimpleTypeJsonUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:161) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:213) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:115) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallList$2(JsonProtocolUnmarshaller.java:144) ~[aws-json-protocol-2.15.67.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_265]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_265]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_265]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_265]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallList(JsonProtocolUnmarshaller.java:146) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:213) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:199) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:169) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:82) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:36) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonResponseHandler.handle(AwsJsonResponseHandler.java:44) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[aws-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$resultTransformationResponseHandler$7(BaseClientHandler.java:269) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.15.67.jar:?]
	... 41 more
Caused by: java.lang.NumberFormatException: For input string: "1.611248009879E9"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_265]
	at java.lang.Long.parseLong(Long.java:589) ~[?:1.8.0_265]
	at java.lang.Long.parseLong(Long.java:631) ~[?:1.8.0_265]
	at software.amazon.awssdk.utils.DateUtils.parseUnixTimestampMillisInstant(DateUtils.java:146) ~[utils-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.lambda$safeParseDate$0(StringToInstant.java:72) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.convert(StringToInstant.java:56) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.core.StringToInstant.convert(StringToInstant.java:32) ~[protocol-core-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$SimpleTypeJsonUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:161) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:213) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:115) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallList$2(JsonProtocolUnmarshaller.java:144) ~[aws-json-protocol-2.15.67.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_265]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_265]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_265]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_265]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_265]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallList(JsonProtocolUnmarshaller.java:146) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:213) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:199) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshall(JsonProtocolUnmarshaller.java:169) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:82) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:36) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonResponseHandler.handle(AwsJsonResponseHandler.java:44) ~[aws-json-protocol-2.15.67.jar:?]
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[aws-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$resultTransformationResponseHandler$7(BaseClientHandler.java:269) ~[sdk-core-2.15.67.jar:?]
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.15.67.jar:?]
	... 41 more```

@dittos dittos mentioned this issue Mar 23, 2021
Open
@danielmewes
Copy link

This appears to be due to ApproximateArrivalTimestamp being expressed as a floating point number (in seconds) in kinesalite, while KCL requires it to be an integer (I believe in milliseconds). Removing the / 1000 from putRecords.js and putRecord.js, avoids the issue for me, but I haven't validated that this is indeed the correct behavior in all cases.

@danielmewes
Copy link

Now wondering if this might actually be caused by aws/aws-sdk-java-v2#1023 in the AWS SDK...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@danielmewes @KKcorps