fix: prevent concatenated gzip response truncation when read via GZIPInputStream

[jencymaryjoseph]

Motivation and Context

When an SDK streaming response body is gzip-compressed and the caller decodes it with
java.util.zip.GZIPInputStream, a concatenated (multi-member) gzip stream can be silently truncated to
only its first member.

Root cause is in the JDK: GZIPInputStream.readTrailer() decides whether another gzip member follows partly
from the underlying stream's available(). A network-backed response body can transiently report
available() == 0 exactly at a member boundary (the socket momentarily has no buffered bytes even though more
data is still in flight). GZIPInputStream treats that transient 0 as end-of-stream, stops decoding, and
drops every remaining member — no error is raised, so the caller just receives partial data.

Modifications

• New internal decorator GzipAvailabilityInputStream (@SdkInternalApi, core/sdk-core):
  • Passively classifies the stream from its first 3 bytes (gzip magic 1f 8b + DEFLATE method 08).
  • For detected gzip content only, reports available() as at least 1 while the stream is open and not at
    EOF, so GZIPInputStream keeps reading across member boundaries instead of stopping on a transient 0.
  • Non-gzip streams pass available() through unchanged, so text/line consumers (e.g. BufferedReader) are
    unaffected.
  • Tracks EOF so the coercion stops at the real end of the stream (terminates, does not hang), propagates
    release() to a Releasable delegate, and preserves read/skip/mark/reset/close semantics.
• Added explicit opt-in support to the blocking-stream transformer factories:
  • ResponseTransformer.toInputStream(boolean)
  • ResponseTransformer.toInputStream(Duration, boolean)
  • AsyncResponseTransformer.toBlockingInputStream(boolean)

The boolean parameter controls GZIPInputStream compatibility. Passing true applies the compatibility wrapper;
passing false preserves the response stream's normal available() behavior. Existing overloads behave as
false, so the change is disabled by default and does not alter existing callers.

For sync requests, the wrapper is applied by ResponseTransformer when it creates the blocking
ResponseInputStream. For async requests, InputStreamResponseTransformer applies it when adapting the response
publisher into a blocking stream. Keeping the setting on the transformer also allows it to survive async
split() processing without client-option propagation or S3-specific decorators.

Testing

• GzipAvailabilityInputStreamTest covers gzip detection, non-gzip pass-through, transient zero availability,
  concatenated and many-member decoding, EOF and close behavior, blocking EOF, skip, mark/reset, release, and
  abort propagation.
• ResponseTransformerToInputStreamTest covers default-off and opt-in behavior, timeout overloads, non-gzip
  streams, abort propagation, and concatenated gzip decoding.
• InputStreamResponseTransformerTest covers default-off and opt-in behavior, non-gzip streams, abort
  cancellation, and preservation of compatibility through async split().
• GetObjectConcatenatedGzipTest verifies opt-in sync and async S3 downloads through the HTTP stack.

SDK Core, AWS Core, and S3 unit and integration tests pass.

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist

• I have read the CONTRIBUTING document
• Local run of mvn install succeeds
• My code follows the code style of this project
• My change requires a change to the Javadoc documentation
• I have updated the Javadoc documentation accordingly
• I have added tests to cover my changes
• All new and existing tests passed
• I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
• My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

• I confirm that this pull request can be released under the Apache 2 license

[dagnir]

Hmm I don't think this is the right approach; it would make more sense to push the decision of whether this is a GZIP inputstream further down the stack (i.e. closer to the transport/HTTP layer). For example, is there some way we can check the content-type header, and if it's GZIP, wrap the stream there?

Another reason this is problematic is what happens if somewhere down the line we need to make another "workaround" for some other type of content stream?

[jencymaryjoseph]

Looked into this further - and I think transport/header gating cannot reliably fix this. S3 ContentType and ContentEncoding(when Content-Encoding: gzip is set, the SDK already decompresses) are user controlled and may not identify gzip. Async transports also expose a Publisher, so available() doesn't exist until the blocking stream adapter that is exactly where ResponseInputStream sits.

Should we keep ResponseInputStream generic by applying GzipAvailabilityInputStream through an internal factory at the sync and async blocking stream boundaries, while reserving ExecutionInterceptor (modifyHttpResponseContent) hooks for future header based workarounds.

[dagnir]

Should we also gate the "always return 1 for GZIP" behavior on something like an SdkAdvancedClientOption? https://docs.aws.amazon.com/java/api/latest/software/amazon/awssdk/core/client/config/SdkAdvancedClientOption.html

Maybe something like V1_COMPATIBLE_GZIP_STREAM_BEHAVIOR? (we can bikeshed on naming). Some customers may not care even if the stream is GZIP because they aren't using GZIPInputStream for example

[jencymaryjoseph]

Updated the fix so the GZIP aware wrapper is applied where the SDK exposes blocking response streams. For sync responses, BaseSyncClientHandler wraps the body before passing it to the customer’s ResponseTransformer. For async responses, InputStreamResponseTransformer applies it when converting the publisher into a blocking ResponseInputStream. This keeps ResponseInputStream generic while preserving non-GZIP behavior, EOF handling, and abort support.

Added a default-on SdkAdvancedClientOption that allows customers to disable this behavior. Sync reads the option in BaseSyncClientHandler, while async configures the transformer before prepare(). S3 applies the disabled setting before multipart and presigned flows split the transformer, including cross-region decorator chains. Tests cover the stream behavior, sync and async handlers, lifecycle ordering, wrapper propagation, multipart, presigned, and cross-region paths

[dagnir]

I think this is the wrong name for this since we're not adding any meaningful support for concatenated gzip data streams; i.e. if you're using something other than GZIPInputStream to wrap the response stream, that doesn't incorrectly close on available() == 0, then there's no issue.

maybe add "compatibility" in the name, like GZIP_INPUTSTREAM_COMPATIBILITY_SUPPORT_ENABLED?

[dagnir]

"The correct behavior is enabled by default" is ambiguous here; does it mean we do or don't return 0 from available()?

[dagnir]

We should do a surface API review for this change and discuss as a team, but I think this behavior should be disabled by default

[dagnir]

This is strangely named; withConcatenatedGzipStreamSupportEnabled sounds like the returned transformer has the GZIP behavior enabled always, but that's not always true and depends on the enabled config

[dagnir]

Same comment here, the name suggests the wrapping always happens but it depends on the config value

[dagnir]

Hmm looking at this makes me rethink my earlier suggestion of pushing the logic out of ResponseTransformer/AsyncResponseTransformer. It's probably worth revisiting; it would simplify things greatly if the configuration were just on the transformer themselves, especially for AsyncResponseTransformer.toFile() since there's nothing you can do at the lower layers to influence the behavior since the InputStream only gets created at the transformer layer.

E.g.

    static <ResponseT> ResponseTransformer<ResponseT, ResponseInputStream<ResponseT>> toInputStream(Duration timeout,
                                                                                                    boolean gzipInputStreamCompatibilitySupportEnabled) {
   ...

[jencymaryjoseph]

Added new blocking stream transformer overloads with a gzipInputStreamCompatibilityEnabled parameter. Passing true applies the GZIP compatibility wrapper; passing false preserves the stream’s normal available() behavior.
Sync now provides toInputStream(boolean) and toInputStream(Duration, boolean), while async provides toBlockingInputStream(boolean). Existing overloads behave as false, so compatibility remains opt-in. This removes the client option, handler propagation and S3 specific decorator logic