Feature Proposal: Pluggable Compression & Decompression Protocols

[o-nnerb]

Feature Proposal: Pluggable Compression & Decompression Protocols

Supersedes #198, which shipped Session.compression(_:) as a closed gzip/deflate enum tied
directly to NIOHTTPRequestCompressor. #196 (Payload Encoding) is unaffected — it already
scoped compression out on purpose ("orthogonal concerns requiring separate protocols") and
stays accurate as-is.

Motivation

The current Compression/Decompression support works, but it's closed: only the two algorithms
RequestDL bundles are possible, and adding a new one means changing this package. The goal is
to let other libraries implement their own compressor/decompressor (Brotli via a C binding,
Zstd, anything) using RequestDL's own request/response pipeline, without inventing new Task
Modifiers or Properties to do it.

Public API

Two protocols, symmetric in shape — both a stateless, Sendable descriptor that manufactures a
stateful, per-request/per-response worker:

public protocol Compressor: Sendable {
    var contentEncodingValue: String { get }
    func callAsFunction() throws -> any CompressorStream
}
public protocol CompressorStream {
    mutating func callAsFunction(compressing bytes: [UInt8]) throws -> [UInt8]
    mutating func finish() throws -> [UInt8]
}

public protocol Decompressor: Sendable {
    var contentEncodingValue: String { get }
    func callAsFunction() throws -> any DecompressorStream
}
public protocol DecompressorStream {
    mutating func callAsFunction(decompressing bytes: [UInt8]) throws -> [UInt8]
    mutating func finish() throws -> [UInt8]
}

The stream methods may return an empty array — a streaming codec is allowed to buffer
internally and only emit once it has enough state, exactly like deflate(). finish() flushes
whatever's left (and is where a truncated/corrupt stream gets caught). An algorithm that
genuinely needs the whole payload before it can decode anything is still a valid, conforming
implementation: buffer everything in callAsFunction, return [] every time, and do the real
work in finish() — it just won't benefit from incremental delivery in DownloadTask.

Built-in algorithms conform to both roles where it applies:

extension Compressor where Self == GzipAlgorithm { public static var gzip: Self { .init() } }
extension Decompressor where Self == GzipAlgorithm { public static var gzip: Self { .init() } }
extension Compressor where Self == DeflateAlgorithm { public static var deflate: Self { .init() } }
extension Decompressor where Self == DeflateAlgorithm { public static var deflate: Self { .init() } }
extension Decompressor where Self == BrotliURLSessionOnlyAlgorithm { public static var brotliURLSessionOnly: Self { .init() } }

GzipAlgorithm/DeflateAlgorithm/BrotliURLSessionOnlyAlgorithm are placeholders — their
callAsFunction throws NativeOnlyAlgorithmError if ever actually invoked. In the normal case
they're never called at all: the OS (URLSession) or async-http-client handles these natively,
and RequestDL just gets out of the way. BrotliURLSessionOnlyAlgorithm is named for what it
is — there's no NIOHTTPCompression brotli decoder anywhere, so it only works under .urlSession,
pure or not.

Where each one attaches

• Decompressor stays on Session (decompressionAlgorithms([.gzip, .deflate])) — it's
  about the response, which exists independent of whether the request has a body at all.
• Compressor moves to Payload/Form via environment (.compression(.gzip)), mirroring
  payloadEncoder. It's specifically about the outgoing body, so it belongs where the body is
  declared, not on Session. This also drops compression/compressionDuplicateHeaderBehavior
  out of Internals.Session.Configuration's pooled-client cache key, which today fragments the
  connection pool for a setting that never actually affects the connection.

Session.DuplicateHeaderBehavior stays compression-only, unchanged in spirit: it resolves a
real conflict (the caller pre-compressed the body themselves and set Content-Encoding
manually vs. RequestDL trying to compress and set it too). Accept-Encoding has no equivalent
legitimate conflict — it's derived entirely from the configured decompressors, full stop.

Executor behavior

URLSession: setting your own Accept-Encoding header switches off CFNetwork's transparent
decoding entirely, for every encoding, not just the one you added — so the request-level
decision is all-or-nothing. If every configured Decompressor is one CFNetwork already handles
natively (gzip/deflate/br), RequestDL stays quiet and lets it decode transparently. The
moment a genuinely custom algorithm is in the list, RequestDL sets Accept-Encoding itself
(listing everything) and decodes all of it manually — including gzip/deflate if they're
also in that same list, since the bypass is total. .disabled now gets real parity with NIO on
.urlSession too, via Accept-Encoding: identity — no more forced fallback to .nio just
because decompression was turned off.

NIO: mirrors the same idea using async-http-client's own native handler instead of
reimplementing it — gzip/deflate in the list translate to
HTTPClient.Configuration.decompression = .enabled(limit:), and NIOHTTPResponseDecompressor
strips Content-Encoding after it decodes, so RequestDL's own manual dispatch (matching
Content-Encoding against the configured list) naturally leaves those alone and only picks up
whatever's left.

Streaming, both directions

Response decompression hooks into Internals.AsyncResponse, the transport-agnostic point where
ResponseHead and the body stream already meet — so DataTask and DownloadTask both get
custom decompression for free. Request compression follows the same shape: RequestBody's
existing chunk-by-chunk streaming feeds a CompressorStream incrementally instead of the
current behavior of buffering the entire body once to compress it as a single blob. The
trade-off: today's exact Content-Length (known upfront because the whole compressed body
already exists before sending) goes away for streamed compression — it falls back to chunked
transfer encoding, which both transports support natively.

Verified locally with a genuinely custom (non-gzip) run-length codec, driven by real network
I/O on both transports in both directions, with chunk boundaries neither client nor server
controlled deliberately made to split encoded pairs mid-pair across reads — reconstructed
correctly every time. Not a synthetic worst case: this is what URLSession.bytes(for:) and
async-http-client's streaming body already do today for gzip, confirmed by timing (first
decompressed bytes arrive within milliseconds of the first compressed bytes on the wire, well
before the response finishes) — the underlying platforms never buffer-then-decode, so neither
should we.

Known limitation

BackgroundDownloadTask never touches Internals.AsyncResponse — it hands the request straight
to URLSessionDownloadTask, mediated by a system daemon that can outlive the app process
itself. Native algorithms are unaffected (confirmed: CFNetwork decodes before the file ever
touches disk). A custom Decompressor can't run mid-transfer here — the only viable path is a
post-download pass over the completed file, decoding it as a discrete step after
didFinishDownloadingTo, before moving it to its final destination.