Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxy protocol: new feature auto-detect proxy protocol #18951

Merged
merged 61 commits into from
May 6, 2022
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
278ce7e
Auto-detect proxy protocol
Nov 9, 2021
121e87d
Fix format
Nov 10, 2021
2d43ae3
Merge branch 'auto_detect_proxy_protocol' of github.com:solo-io/envoy…
Nov 10, 2021
de6b7f2
Fix format
Nov 10, 2021
3a36695
Merge branch 'main' into auto_detect_proxy_protocol
Nov 10, 2021
409f94c
Fix merge
Nov 10, 2021
ae19e1f
Clearer error message
Nov 10, 2021
ed4fb54
Add unit test for partial v1 protocol
Nov 10, 2021
bcc7aef
Rename new field to
Nov 10, 2021
8c87bcd
Fix format
Nov 10, 2021
4aa520e
Minor tweak to version history note
Nov 10, 2021
91c9a1b
Use attention clause in API comment
Nov 10, 2021
c45ecc7
Set in ctor initialization list
Nov 10, 2021
4259f1a
Add comment to clarify test
Nov 10, 2021
2b4a89a
Return value was only used in one of two return paths, tiny refactor
Nov 10, 2021
f105c3d
Fix format
Nov 10, 2021
6a0d49b
Test case with partial signature, just char
Nov 10, 2021
f47232f
Use standard library for min
Nov 10, 2021
fa9746a
PR comments; fix bug with large requests being allowed
Nov 10, 2021
a37ad43
Assert msg size in tests in addition to noting in comment
Nov 10, 2021
6f19883
Do not allow any false positives
Nov 10, 2021
0927000
Only check if proxy protocol on initial MSG_PEEK
Nov 11, 2021
b5cbfc1
Don't check too many bytes on v1
Nov 11, 2021
a7ef68f
Revert "Don't check too many bytes on v1"
Nov 11, 2021
9495853
Merge branch 'main' into auto_detect_proxy_protocol
Dec 16, 2021
ffb56d1
Merge branch 'main' into auto_detect_proxy_protocol
Apr 28, 2022
ac2982f
Fix bad merge on current.rst
Apr 28, 2022
2a66a01
Formatting
Apr 28, 2022
c9e086f
Wait for more bytes if we can't tell
Apr 29, 2022
af6f88d
Update API comment to reflect waiting behavior
Apr 29, 2022
7f17660
Formatting
Apr 29, 2022
59206da
ReadOrParseState is not always an error if not Done
Apr 29, 2022
b416242
Improve comments
Apr 29, 2022
4e4638a
Formatting
Apr 29, 2022
354a365
Better test for waiting for more bytes
Apr 29, 2022
86ad288
More accurate comment
Apr 29, 2022
899f199
We can wait for less bytes for v1 proxy protocol
Apr 29, 2022
887d0bc
Fix spelling error
Apr 30, 2022
a23ac5f
Reuse previous parsing logic; split checks on allowRequestsWithoutPro…
Apr 30, 2022
6075b67
Revert "Reuse previous parsing logic; split checks on allowRequestsWi…
Apr 30, 2022
4d6f75f
Add v2 partial tests
Apr 30, 2022
2cb4526
Reuse existing parsing logic
May 2, 2022
1667461
Remove whitespace in test
May 2, 2022
a0c977f
Move check to else statement
May 2, 2022
85706e6
Clarify what tests are covering
May 2, 2022
c86edef
Add another test and fix for v2 followed by v1 signature bytes
May 2, 2022
6031151
Add tests to ensure we disconnect invalid requests after valid signature
May 2, 2022
a09bb1b
Use capital letters
May 2, 2022
cfeda5a
Use capital letters
May 2, 2022
5a660a0
Use const
May 2, 2022
e92a8a4
Do all checking in one place up front
May 2, 2022
84cdc32
Do all checking in one place up front; remove unneeded check
May 2, 2022
f3d65e9
Merge branch 'main' into auto_detect_proxy_protocol
May 2, 2022
b2d467e
Merge branch 'main' into auto_detect_proxy_protocol
May 3, 2022
45e9c24
Merge branch 'main' into auto_detect_proxy_protocol
May 3, 2022
056f74f
Add changelog
May 3, 2022
39e7b50
Remove current.rst file that was added in bad merge
May 3, 2022
392124e
PR comments
May 4, 2022
307df1c
Merge branch 'main' into auto_detect_proxy_protocol
May 4, 2022
5c50e23
Ensure every case is handled in switch statement
May 5, 2022
36bbfa2
Merge branch 'main' into auto_detect_proxy_protocol
May 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,14 @@ message ProxyProtocol {

// The list of rules to apply to requests.
repeated Rule rules = 1;

// Allow requests through that don't use proxy protocol. Defaults to false.
//
// .. attention::
//
// Only enable if ALL traffic to the listener comes from a trusted source.
// For more information on the security implications of this feature, see
// https://www.haproxy.org/download/1.9/doc/proxy-protocol.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the doc comment a couple lines up, it references version 2.1's copy of the doc. Please use the same url as that one (or update both of them), so the user isn't left wondering if there's a significant difference between the two links.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in fa9746a

//
bool allow_requests_without_proxy_protocol = 2;
}
1 change: 1 addition & 0 deletions docs/root/version_history/current.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ New Features
* listener: added support for :ref:`MPTCP <envoy_v3_api_field_config.listener.v3.Listener.enable_mptcp>` (multipath TCP).
* oauth filter: added :ref:`cookie_names <envoy_v3_api_field_extensions.filters.http.oauth2.v3.OAuth2Credentials.cookie_names>` to allow overriding (default) cookie names (``BearerToken``, ``OauthHMAC``, and ``OauthExpires``) set by the filter.
* oauth filter: setting IdToken and RefreshToken cookies if they are provided by Identity provider along with AccessToken.
* proxy protocol: added support for allowing requests without proxy protocol on the listener from trusted downstreams as an opt-in flag.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a link to the new config setting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 392124e

* tcp: added a :ref:`FilterState <envoy_v3_api_msg_type.v3.HashPolicy.FilterState>` :ref:`hash policy <envoy_v3_api_msg_type.v3.HashPolicy>`, used by :ref:`TCP proxy <envoy_v3_api_field_extensions.filters.network.tcp_proxy.v3.TcpProxy.hash_policy>` to allow hashing load balancer algorithms to hash on objects in filter state.
* tcp_proxy: added support to populate upstream http connect header values from stream info.
* thrift_proxy: add header to metadata filter for turning headers into dynamic metadata.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ namespace ProxyProtocol {
Config::Config(
Stats::Scope& scope,
const envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol& proto_config)
: stats_{ALL_PROXY_PROTOCOL_STATS(POOL_COUNTER(scope))} {
: stats_{ALL_PROXY_PROTOCOL_STATS(POOL_COUNTER(scope))},
allow_requests_without_proxy_protocol_(proto_config.allow_requests_without_proxy_protocol()) {
for (const auto& rule : proto_config.rules()) {
tlv_types_[0xFF & rule.tlv_type()] = rule.on_tlv_present();
}
Expand All @@ -63,6 +64,10 @@ const KeyValuePair* Config::isTlvTypeNeeded(uint8_t type) const {

size_t Config::numberOfNeededTlvTypes() const { return tlv_types_.size(); }

bool Config::allowRequestsWithoutProxyProtocol() const {
return allow_requests_without_proxy_protocol_;
}

Network::FilterStatus Filter::onAccept(Network::ListenerFilterCallbacks& cb) {
ENVOY_LOG(debug, "proxy_protocol: new connection accepted");
Network::ConnectionSocket& socket = cb.socket();
Expand All @@ -77,11 +82,19 @@ Network::FilterStatus Filter::onAccept(Network::ListenerFilterCallbacks& cb) {
return Network::FilterStatus::StopIteration;
}

void Filter::resetAndContinue(Network::IoHandle& io_handle) {
// Release the file event so that we do not interfere with the connection read events.
io_handle.resetFileEvents();
cb_->continueFilterChain(true);
}

void Filter::onRead() {
const ReadOrParseState read_state = onReadWorker();
if (read_state == ReadOrParseState::Error) {
config_->stats_.downstream_cx_proxy_proto_error_.inc();
cb_->continueFilterChain(false);
} else if (read_state == ReadOrParseState::SkipFilterError) {
resetAndContinue(cb_->socket().ioHandle());
}
}

Expand Down Expand Up @@ -135,9 +148,7 @@ ReadOrParseState Filter::onReadWorker() {
proxy_protocol_header_.value().remote_address_);
}

// Release the file event so that we do not interfere with the connection read events.
socket.ioHandle().resetFileEvents();
cb_->continueFilterChain(true);
resetAndContinue(socket.ioHandle());
return ReadOrParseState::Done;
}

Expand Down Expand Up @@ -309,7 +320,7 @@ ReadOrParseState Filter::parseExtensions(Network::IoHandle& io_handle, uint8_t*
const auto recv_result = io_handle.recv(buf, to_read, 0);
if (!recv_result.ok()) {
if (recv_result.err_->getErrorCode() == Api::IoError::IoErrorCode::Again) {
return ReadOrParseState::TryAgainLater;
return ReadOrParseState::TryAgainLaterError;
}
ENVOY_LOG(debug, "failed to read proxy protocol (no bytes avail)");
return ReadOrParseState::Error;
Expand Down Expand Up @@ -424,7 +435,7 @@ ReadOrParseState Filter::readProxyHeader(Network::IoHandle& io_handle) {

if (!result.ok()) {
if (result.err_->getErrorCode() == Api::IoError::IoErrorCode::Again) {
return ReadOrParseState::TryAgainLater;
return ReadOrParseState::TryAgainLaterError;
}
ENVOY_LOG(debug, "failed to read proxy protocol (no bytes read)");
return ReadOrParseState::Error;
Expand All @@ -434,6 +445,13 @@ ReadOrParseState Filter::readProxyHeader(Network::IoHandle& io_handle) {
if (nread < 1) {
ENVOY_LOG(debug, "failed to read proxy protocol (no bytes read)");
return ReadOrParseState::Error;
} else if (nread < PROXY_PROTO_V2_HEADER_LEN &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this handles the case where there is no PROXY header, but the number of bytes read is larger than PROXY_PROTO_V2_HEADER_LEN, eg the filter reads an entire HTTP/1 request header.

Copy link
Contributor Author

@kdorosh kdorosh Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, thanks! addressed in fa9746a

i went ahead and wrote a test that failed (the new one), then removed the faulty logic. now there are two tests (for small and large requests that get passed through the filter) that hit the exact same code path. happy to remove either one, what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave both tests in

config_.get()->allowRequestsWithoutProxyProtocol()) {
if (nread < PROXY_PROTO_V1_SIGNATURE_LEN ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although it is rare case, there still have the chance of receiving the partial v1 signature, like just a 'P' char, the left of the signature received by the second recv call. But this check will just skip the left of signature checking.

we could return SkipFilterError at https://github.com/envoyproxy/envoy/pull/18951/files#diff-a35fc10b1ce97239d02ac1791aac05069eaf253d8efc77cefe93749f901e36abL446
Then it will got all the data, and compared the signature of v1 and v2.

But if we only receive very short data from the client (short than both v1 and v2 signature), then I feel we only can waiting for a timeout of listener filter. But that is bad for the usecase of issue.

Another idea is if the nread is less than v1 and v2 signature, then we compare part of signature, like if the first byte is X', then compare to the first byte of v1 signature, it is P`. Also compare the first byte of v2 signature, then we know it can't be the proxy protocol.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soulxu can you update the link from

we could return SkipFilterError at https://github.com/envoyproxy/envoy/pull/18951/files#diff-a35fc10b1ce97239d02ac1791aac05069eaf253d8efc77cefe93749f901e36abL446
Then it will got all the data, and compared the signature of v1 and v2.

to a link to main / a commit and where you're proposing this change? I think with the recent commits to this PR that the link provided is dated, I'm not sure where you're proposing we return the error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also related #18951 (comment)

memcmp(buf_, PROXY_PROTO_V1_SIGNATURE, PROXY_PROTO_V1_SIGNATURE_LEN)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be
((nread < PROXY_PROTO_V2_SIGNATURE_LEN || memcmp(buf_, PROXY_PROTO_V2_SIGNATURE, nread)) && memcmp(buf_, PROXY_PROTO_V1_SIGNATURE, nread))

Copy link
Contributor Author

@kdorosh kdorosh Nov 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is actually fine as is. the first if statement in this block ensures that by the time we get here we know nread >= PROXY_PROTO_V1_SIGNATURE_LEN. then in this if statement the nread < PROXY_PROTO_V2_SIGNATURE_LEN case short-circuits if needed, so in the second memcmp we know nread >= PROXY_PROTO_V2_SIGNATURE_LEN. in short, we know nread is >= both signature lengths before we do memcmp with the signature, and we want to check at most the signature length

if it helps with readability I'm happy to add std::min on nread and the signature lengths, but I do not think there is a logic error here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, you are right.

I was based on the logic we want to exam more bytes.

ENVOY_LOG(debug, "need more bytes to determine if we have a proxy protocol header");
return ReadOrParseState::SkipFilterError;
}
}

if (buf_off_ + nread >= PROXY_PROTO_V2_HEADER_LEN) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,22 @@ class Config : public Logger::Loggable<Logger::Id::filter> {
*/
size_t numberOfNeededTlvTypes() const;

/**
* Filter configuration that determines if we should pass-through requests without
* proxy protocol. Should only be configured to true for trusted downstreams.
*/
bool allowRequestsWithoutProxyProtocol() const;

private:
absl::flat_hash_map<uint8_t, KeyValuePair> tlv_types_;
bool allow_requests_without_proxy_protocol_{};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const

};

using ConfigSharedPtr = std::shared_ptr<Config>;

enum ProxyProtocolVersion { Unknown = -1, InProgress = -2, V1 = 1, V2 = 2 };

enum class ReadOrParseState { Done, TryAgainLater, Error };
enum class ReadOrParseState { Done, TryAgainLaterError, Error, SkipFilterError };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you rename TryAgainLater?

I think SkipFilterError should be renamed to SkipFilter. It isn't an error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in fa9746a


/**
* Implementation the PROXY Protocol listener filter
Expand Down Expand Up @@ -98,7 +105,7 @@ class Filter : public Network::ListenerFilter, Logger::Loggable<Logger::Id::filt
/**
* Helper function that attempts to read the proxy header
* (delimited by \r\n if V1 format, or with length if V2)
* @return bool true valid header, false if more data is needed or socket errors occurred.
* @return ReadOrParseState Done if successfully parsed, or the error type.
*/
ReadOrParseState readProxyHeader(Network::IoHandle& io_handle);

Expand All @@ -118,6 +125,8 @@ class Filter : public Network::ListenerFilter, Logger::Loggable<Logger::Id::filt
bool parseV2Header(char* buf);
absl::optional<size_t> lenV2Address(char* buf);

void resetAndContinue(Network::IoHandle& io_handle);

Network::ListenerFilterCallbacks* cb_{};

// The offset in buf_ that has been fully read
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,19 @@ TEST_P(ProxyProtocolTest, V1Basic) {
disconnect();
}

TEST_P(ProxyProtocolTest, AllowNoProxyProtocol) {
// allows request through even though it doesn't use proxy protocol
envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config;
proto_config.set_allow_requests_without_proxy_protocol(true);
connect(true, &proto_config);

write("more data");

expectData("more data");

disconnect();
}

TEST_P(ProxyProtocolTest, V1Minimal) {
connect();
write("PROXY UNKNOWN\r\nmore data");
Expand Down Expand Up @@ -927,6 +940,30 @@ TEST_P(ProxyProtocolTest, PartialRead) {
disconnect();
}

TEST_P(ProxyProtocolTest, PartialV1ReadWithAllowNoProxyProtocol) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also worth testing a case that we have correct v1 or v2 header signature, but invalid data in the rest of header, then ensure we disconnect the connection, and not let the connection passthrough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 6031151


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, useless space line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 1667461

envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config;
proto_config.set_allow_requests_without_proxy_protocol(true);
connect(true, &proto_config);

write("PROXY TCP4");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should test the case of the first write is not full v1 protocol signature, like just one char 'P'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6a0d49b

Copy link
Contributor Author

@kdorosh kdorosh Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure this is possible simultaneously with the fragmented/partial protocol without the possibility for false positive, e.g.

TEST_P(ProxyProtocolTest, TinyPartialV1ReadWithAllowNoProxyProtocolThisFails) {

  envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config;
  proto_config.set_allow_requests_without_proxy_protocol(true);
  connect(true, &proto_config);

  write("P"); // matches the beginning of v1 proxy protocol

  dispatcher_->run(Event::Dispatcher::RunType::NonBlock);

  write("BOGUS");

  expectData("PBOGUS"); // we will consume `P` and upstream gets `BOGUS` not `PBOGUS`

  disconnect();
}

this is why in the initial implementation I required that the initial read from recv have enough bytes available on the initial MSG_PEEK to make the determination on the v1/v2 proxy header.

We can leave the PR as it is (or perhaps preferably throw a hard error if we consumed some bytes and then realize we had a false positive?) with the understanding that there could be false positives on identifying proxy protocol, additionally with the understanding that this is quite rare and will not likely happen in practice (except perhaps for single byte reads of P and HTTP POST/PUT request?). In exchange, we get support for fragmented/partial reads. Thoughts?

Also willing to go back to my original version (before 6a0d49b which addresses the initial concern with partial reads) which addresses the potential false positive here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not following. How would the filter consume the P only?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running the test above, we do recv with MSG_PEEK and just get P here.

Then before we continue the loop, we read that byte for real (no MSG_PEEK) here. Thus when we do our second recv and we get BOGUS we have already consumed P. We correctly identify that the request is not proxy protocol, but the request forwarded upstream has a missing byte(s) (in this case, just P).

The only way I see to avoid false positives identifying the header and always send the correct response up the filter chain is to require that the initial MSG_PEEK has enough bytes to detect v1/v2 header (at most 16, which seems reasonable for this opt-in only codepath)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see.

I think you need to fix this case. You may have to restructure the code such that no bytes are read (without MSG_PEEK) until we've decided 100% whether this is a proxy header or not.

I think it's probably good enough to look for either PROXY_PROTO_V1_SIGNATURE or PROXY_PROTO_V2_SIGNATURE. If the connections starts with either of those, but then doesn't have a valid full proxy protocol header, abort the connection as we currently do. If the connection starts with bytes that match neither of those signatures, and the new option is enabled, don't read() any bytes and continue the filter chain. Will that work?

Copy link
Contributor Author

@kdorosh kdorosh Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will work. Implemented in 6f19883 and 0927000


dispatcher_->run(Event::Dispatcher::RunType::NonBlock);

write(" 254.254.2");
write("54.254 1.2");
write(".3.4 65535");
write(" 1234\r\n...");

expectData("...");

EXPECT_EQ(server_connection_->connectionInfoProvider().remoteAddress()->ip()->addressAsString(),
"254.254.254.254");
EXPECT_TRUE(server_connection_->connectionInfoProvider().localAddressRestored());

disconnect();
}

TEST_P(ProxyProtocolTest, V2PartialRead) {
// A well-formed ipv4/tcp header, delivered with part of the signature,
// part of the header, rest of header + body
Expand Down