incomplete stream from S3Client getObject

[KnutKnutson]

I'm attempting to stream a file of concatenated gzip files from s3 (~100mb in size) using version 2.20.109 of the sdk

The kotlin code I'm using looks like

val request = GetObjectRequest.builder()
    .bucket(bucketName)
    .key(key)
    .build()
val resp: ResponseInputStream<GetObjectResponse> = client.getObject(request)
GZIPInputStream(resp).bufferedReader().useLines { sequence ->
  sequence.iterator().forEachRemaining { line ->
    doSomething(line)
  }
}

However when iterating the lines, the iterator is stopped (hasNext returns false) before all the lines have been read.
The number of lines read is inconsistent for the same file. Sometimes 18 lines, sometimes 32 or 34, etc.

Does anyone have any guidance on what might be the issue?
Thanks

[polyomino24]

I encountered the same problem, but only for large gzip files (ten thousand lines or more).

[polyomino24]

Hi @KnutKnutson .
I found that root cause is java.util.zip.GZIPInputStream in my case.
Maybe these pages help you.

• https://stackoverflow.com/questions/41400810/gzipinputstream-closes-prematurely-when-decompressing-httpinputstream
• https://pzampino.github.io/2018/10/09/gzipinputstream-and-available.html

In my case, changing GZIPInputStream to Apache Commons GzipCompressorInputStream with decompressConcatenated=true option worked fine.