refactor(http/upgrade): `Http11Upgrade` is `Clone` #3540

cratelyn · 2025-01-17T15:21:04Z

in the 1.0 interface of the http crate, request and response extensions must
now satisfy a Clone bound. Http11Upgrade was written before this was the
case, and is very intentionally designed around the idea that it not be
cloneable.

insert_half() in particular could cause the proxy to panic if it were
to clone a request or response's extensions. it might call
insert_half() a second time, and discover that the TryLock<T> had
already been set.

moreover, holding on to a copy of the extensions would prevent the
Drop method for Inner from being called. This would cause
connections that negotiate an HTTP/1.1 upgrade to deadlock due to the
OnUpgrade futures never being polled, and failing to create a Duplex
that acts as the connection's I/O transport.

this commit makes use of the alterations to Http11Upgrade made in
previous commits, and adds a safe implementation of Clone. by
only shallowly copying the extension, we tie the upgrade glue to a
specific request/response.

the extension can be cloned, but any generated copies will be inert.

linkerd/http/upgrade/src/upgrade.rs

this is a noöp change, to set the stage for subsequent changes to the internal model of `Http11Upgrade`. this `inner` field will shortly be an option, and this will make it easier to only follow these panicking branches when the inner lock is `Some(_)`. Signed-off-by: katelyn martin <[email protected]>

this commit hinges on this change to the upgrade middleware's `inner` field. we still retain a reference-counted copy of the `Inner` state, but now we may store `None` here. ``` pub struct Http11Upgrade { half: Half, - inner: Arc<Inner>, + inner: Option<Arc<Inner>>, } ``` a new branch is added to the `insert_half` method that consumes the "sender" and inserts an upgrade future; when this is `None` it will do nothing, rather than panicking. Signed-off-by: katelyn martin <[email protected]>

this type is an empty flag to indicate whether an `Http11Upgrade` extension corresponds to the server or client half of the upgrade future channel. this type is made copy, to facilitate making the `Http11Upgrade` extension safely cloneable. Signed-off-by: katelyn martin <[email protected]>

this commit makes `Http11Upgrade` a cloneable type. see <linkerd/linkerd2#8733>. in the 1.0 interface of the `http` crate, request and response extensions must now satisfy a `Clone` bound. `Http11Upgrade` was written before this was the case, and is very intentionally designed around the idea that it *not* be cloneable. `insert_half()` in particular could cause the proxy to panic if it were to clone a request or response's extensions. it might call `insert_half()` a second time, and discover that the `TryLock<T>` had already been set. moreover, holding on to a copy of the extensions would prevent the `Drop` method for `Inner` from being called. This would cause connections that negotiate an HTTP/1.1 upgrade to deadlock due to the `OnUpgrade` futures never being polled, and failing to create a `Duplex` that acts as the connection's I/O transport. this commit makes use of the alterations to `Http11Upgrade` made in previous commits, and adds a *safe* implementation of `Clone`. by only shallowly copying the extension, we tie the upgrade glue to a *specific* request/response. the extension can be cloned, but any generated copies will be inert. Signed-off-by: katelyn martin <[email protected]>

olix0r

and doc lints, as you noted...

linkerd/http/upgrade/src/upgrade.rs

Signed-off-by: katelyn martin <[email protected]>

`FutureExt::map_ok()` won't work if we try to return an error from this block. the `and_then()` adaptor is used to chain futures, and also won't work given a synchronous closure. this can be done with the equivalent `.await` syntax, and leaves a nicer hole for us to propagate other errors here, shortly. Signed-off-by: katelyn martin <[email protected]>

#3540 (comment) Signed-off-by: katelyn martin <[email protected]> Co-Authored-By: Oliver Gould <[email protected]>

Signed-off-by: katelyn martin <[email protected]>

note: this commit will not compile, code changes are intentionally elided from this commit. this commit upgrades hyper, http, tonic, prost, related dependencies, and their assorted cargo features. see <linkerd/linkerd2#8733>. see also: * #3379 * #3380 * #3382 * #3405 * hyperium/hyper#3796 * #3411 * #3421 * #3427 * #3428 * #3432 * #3433 * #3444 * #3445 * #3454 * #3455 * #3456 * #3457 * #3461 * #3459 * #3465 * #3466 * #3467 * #3468 * linkerd/linkerd2-proxy-api#421 * linkerd/linkerd2#13492 * linkerd/linkerd2#13493 * hyperium/hyper#3816 * #3472 * #3473 * #3479 * tokio-rs/tokio#7059 * #3509 * hyperium/http-body#140 * #3515 * hyperium/http-body#141 * #3530 * #3531 * #3540 * #3556 * #3558 * #3559 * #3564 * #3567 * #3573 * #3583 * hyperium/http-body#144 * #3585 * #3586 * #3597 * #3598 * #3611 * #3614 * #3615 * #3616 * #3647 * #3651 * #3653 * #3654 * #3655 * #3656 * #3657 * #3660 * #3671 * #3672 * #3673 * #3676 * hyperium/http-body#147 * #3692 * #3699 * #3700 * #3701 * #3708 * linkerd/drain-rs#36 * #3715 * #3717 * eminence/procfs#340 --- squash: chore(deps): add hyper-util workspace dependency chore(deps): add http-body-util workspace dependency chore(deps): upgrade linkerd2-proxy-api this commit represents main as of linkerd/linkerd2-proxy-api#421. Signed-off-by: katelyn martin <[email protected]>

## ✨ chore(deps): upgrade to hyperium 1.x crates this branch performs an exciting upgrade for our proxy. this branch upgrades a number of our dependencies so that we use the 1.0 release family of the `hyper` http framework, and its ecosystem. see the [v1.0 announcement][hyper-v1] for more information. this branch upgrades the following dependencies: * `h2`: 0.3 -> 0.4 * `http`: 0.2 -> 1 * `http-body`: 0.4 -> 1 * `hyper`: 0.14.32 -> 1 * `prost`: 0.12 -> 0.13 * `prost-build`: 0.12 -> 0.13 * `prost-types`: 0.12 -> 0.13 * `tonic`: 0.10 -> 0.12 * `tonic-build`: 0.10 -> 0.12 a `hyper-util` dependency is added, which provides among other things, legacy-compatible interfaces such as `hyper_util::client::legacy::Client`, or glue to use `hyper` with the tokio runtime. see <https://docs.rs/hyper-util/latest/hyper_util/> for more information. a `http-body-util` dependency is added, which provides a `BodyExt` trait and a channel-backed body for use in unit tests. the `deprecated` feature flag that was active on our `0.14` hyper dependency has been removed, along with the `stream` and `runtime` feature flags. the `linkerd2-proxy-api` dependency is updated. see: <linkerd/linkerd2-proxy-api#421> ### 📝 notes for review bear particular attention to changes involving `http_body::Body` middleware. the change from two separate `poll_data()` and `poll_trailers()` functions, to a single `poll_frame()` method, induces some subtle changes to various pieces of middleware. also bear in mind that failing to set a timer, in our case `hyper_util::rt::TokioTimer`, can cause http/2 clients, or http/1 and http/2 servers, to panic. make sure that any uses of `hyper::server::conn::http1::Builder`, `hyper::client::conn::http1::Builder`, or `hyper::client::conn::http2::Builder` install a timer. ### ❗ breaking change: `l5d-proxy-error` values the `l5d-proxy-error` header can be examined to observe the cause of proxy errors encountered when sending meshed traffic. by virtue of this using a newer `hyper` client in the proxy, some error messages may in turn look different. for example, an error like `"connect timed out after 1s"` may now appear as `"client error (Connect)"`. ### 📚 other notes this work, by virtue of touching so many parts of the system, is carried out in distinct commits. an initial commit upgrades the dependencies at th workspace level. subsequent commits will not compile if the `--workspace` flag is provided, but the intent of this branch is to update each crate individually. use commands like, e.g. `cargo check --tests -p linkerd-proxy-http` to build particular crates at intermediate commits within this branch. this commit is also only the final leaf in an _extended_ line of work. this has been done to mitigate the effort of reviewing this change, and the risk of churn in the event of any unanticipated errors. see the top-level comment in linkerd/linkerd2#8733 for an overview of all of the work that brought us to this juncture. [hyper-v1]: https://seanmonstar.com/blog/hyper-v1/ --- * chore(deps): upgrade to hyper 1.x note: this commit will not compile, code changes are intentionally elided from this commit. this commit upgrades hyper, http, tonic, prost, related dependencies, and their assorted cargo features. see <linkerd/linkerd2#8733>. see also: * #3379 * #3380 * #3382 * #3405 * hyperium/hyper#3796 * #3411 * #3421 * #3427 * #3428 * #3432 * #3433 * #3444 * #3445 * #3454 * #3455 * #3456 * #3457 * #3461 * #3459 * #3465 * #3466 * #3467 * #3468 * linkerd/linkerd2-proxy-api#421 * linkerd/linkerd2#13492 * linkerd/linkerd2#13493 * hyperium/hyper#3816 * #3472 * #3473 * #3479 * tokio-rs/tokio#7059 * #3509 * hyperium/http-body#140 * #3515 * hyperium/http-body#141 * #3530 * #3531 * #3540 * #3556 * #3558 * #3559 * #3564 * #3567 * #3573 * #3583 * hyperium/http-body#144 * #3585 * #3586 * #3597 * #3598 * #3611 * #3614 * #3615 * #3616 * #3647 * #3651 * #3653 * #3654 * #3655 * #3656 * #3657 * #3660 * #3671 * #3672 * #3673 * #3676 * hyperium/http-body#147 * #3692 * #3699 * #3700 * #3701 * #3708 * linkerd/drain-rs#36 * #3715 * #3717 * eminence/procfs#340 --- squash: chore(deps): add hyper-util workspace dependency chore(deps): add http-body-util workspace dependency chore(deps): upgrade linkerd2-proxy-api this commit represents main as of linkerd/linkerd2-proxy-api#421. Signed-off-by: katelyn martin <[email protected]> * chore(http/box): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(hyper-balance): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/retain): ugrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/stream-timeouts): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/classify): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/upgrade): upgrade to hyper 1.x NOTE: there is a comment noting that the upgrade middleware does not expect to be cloneable. it is unfortunately, however, at odds with the new bounds expected of extensions. so, `Http11Upgrade` is now Clone'able, but a comment is left in place noting this weakened invariant. it's worth investigating how upgrades have changed since, in more detail, but for the current moment we are interested in being especially conservative about changing behavior, and focusing on api changes like `Body::poll_frame(..)`. Signed-off-by: katelyn martin <[email protected]> * chore(metrics): upgrade to hyper 1.x a brief note; this commit happened to tickle an unfortunate sharp edge in `BoxBody` and `Full`'s respective constructors. type inference could not figure out how to construct the body, so we refrain from boxing the response body now. Signed-off-by: katelyn martin <[email protected]> * chore(http/metrics): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/prom): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/insert): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/retry): deprecate linkerd-http-body-compat Signed-off-by: katelyn martin <[email protected]> * chore(mock/http-body): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(http/retry): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(proxy/tap): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(proxy/http): update to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/core): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/test): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/admin): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/outbound): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/inbound): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app/integration): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(app): upgrade to hyper 1.x Signed-off-by: katelyn martin <[email protected]> * chore(transport-header): update generated code Signed-off-by: katelyn martin <[email protected]> * chore(spiffe-proto): update generated code Signed-off-by: katelyn martin <[email protected]> * chore(opencensus-proto): update generated code Signed-off-by: katelyn martin <[email protected]> * chore(opentelemetry-proto): update generated code Signed-off-by: katelyn martin <[email protected]> * chore(deny.toml): update cargo-deny directives this commit updates the contents of `deny.toml`. Signed-off-by: katelyn martin <[email protected]> * chore: `compile` has been renamed to `compile_protos` this addresses deprecation warnings, updating calls to a function that has since been renamed. Signed-off-by: katelyn martin <[email protected]> * chore(deps): remove `linkerd-http-body-compat` dependencies this commit removes this crate, which we added to future proof code for this upgrade, from its dependents. Signed-off-by: katelyn martin <[email protected]> * chore(http/body-compat): remove `linkerd-http-body-compat` crate Signed-off-by: katelyn martin <[email protected]> * chore(deps): update to drain 0.2.1 see linkerd/drain-rs#41. Signed-off-by: katelyn martin <[email protected]> --------- Signed-off-by: katelyn martin <[email protected]>

cratelyn mentioned this pull request Jan 15, 2025

proxy: Figure out the plan for Hyper 1.0 migration linkerd/linkerd2#8733

Open

cratelyn marked this pull request as ready for review January 17, 2025 15:31

cratelyn requested a review from a team as a code owner January 17, 2025 15:31

cratelyn commented Jan 17, 2025

View reviewed changes

linkerd/http/upgrade/src/upgrade.rs Show resolved Hide resolved

olix0r self-assigned this Jan 20, 2025

cratelyn added 4 commits January 20, 2025 20:45

cratelyn force-pushed the kate/http-1.1-upgrade-middleware-is-clone branch from 2231b8b to 476aa04 Compare January 21, 2025 01:46

olix0r reviewed Jan 21, 2025

View reviewed changes

linkerd/http/upgrade/src/upgrade.rs Outdated Show resolved Hide resolved

cratelyn and others added 5 commits January 22, 2025 09:49

chore(http/upgrade): fix broken intradoc links

ba9b6c4

Signed-off-by: katelyn martin <[email protected]>

chore(http/upgrade): add thiserror dependency

f98c689

Signed-off-by: katelyn martin <[email protected]>

review(http/upgrade): propagate insert_half() failures

59d31d5

#3540 (comment) Signed-off-by: katelyn martin <[email protected]> Co-Authored-By: Oliver Gould <[email protected]>

docs(http/upgrade): tweak comment

97557a6

Signed-off-by: katelyn martin <[email protected]>

olix0r approved these changes Jan 22, 2025

View reviewed changes

cratelyn merged commit 76b5f10 into main Jan 22, 2025
15 checks passed

cratelyn deleted the kate/http-1.1-upgrade-middleware-is-clone branch January 22, 2025 17:24

cratelyn mentioned this pull request Mar 18, 2025

chore(deps)!: upgrade to hyper 1.x #3504

Merged

cratelyn mentioned this pull request Mar 18, 2025

refactor(http/upgrade): remove HttpConnect extension #3779

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(http/upgrade): `Http11Upgrade` is `Clone` #3540

refactor(http/upgrade): `Http11Upgrade` is `Clone` #3540

Uh oh!

cratelyn commented Jan 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

olix0r left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

refactor(http/upgrade): Http11Upgrade is Clone #3540

refactor(http/upgrade): Http11Upgrade is Clone #3540

Uh oh!

Conversation

cratelyn commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

olix0r left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

refactor(http/upgrade): `Http11Upgrade` is `Clone` #3540

refactor(http/upgrade): `Http11Upgrade` is `Clone` #3540

cratelyn commented Jan 17, 2025 •

edited

Loading