Separate repository structure for Envoy core and extensions

[htuch]

One of the significant contributors to Envoy’s continued dependency and SLOC growth is C++ extensions. Visualizations of this are provided in “Understanding, maintaining and securing Envoy's supply chain” EnvoyCon 2020 talk (slides, video).

Many of these extensions have limited support, maintenance and production use. Some are labeled alpha or have unknown security postures (see this list). One reason for the growth in C++ extensions is that Envoy has an unstable API internal C++ API, and upstreaming an extension ensures build maintenance as this API changes.

To structurally eliminate the dependencies implied by these extensions, we can ask that new external dependencies that are immature, not widely used or that are not in some sense “core” to Envoy, be added to either:

1. An independent extension repository.
2. A contrib/ directory tree in the Envoy repository that is distinct from the rest of the Envoy source.

This extension repository/directory will be validated for build/test by CI, but will not be covered by Envoy’s security policy or product security team. When an extension meets specified maturity criteria, it can graduate from the extension repository to the regular source tree. Proposed criteria:

• The extension is stable (not alpha)
• The extension’s dependencies meet all Envoy’s dependency criteria
• The extension is known to be in production use by at least one known operator
• The extension has active maintainers
• The Envoy security team agrees that there is a compelling public interest in supporting the extension

There are also considerations and advantages around build time and binary size that might argue in favor of this approach.

A counter-argument to repository separation is that it is burdensome to work across multiple repositories for a typical git flow and we can effectively enforce extension support and dependency separation in a single repository with the above criteria being used to eliminate extensions and dependencies from security team consideration when unmet.

We might also have different standards for API and code review in the extension repository/directory, making maintainership more scalable.

Opening this issue to track discussion on pros/cons of this approach.

CC @envoyproxy/maintainers @envoyproxy/security-team

[lizan]

From CI perspective, IMO it is hard to support a separate repository and that reduces the product velocity, instead we might just do another directory (e.g. contrib) with different set of bazel workspace / build rules?

[htuch]

This makes sense to me; it's challenging as a developer flow as well to have multiple repos. I like the idea of contrib and then having a clear path to migration to source/extensions based on specific criteria. I think by default we would not include contrib in the Envoy binaries built, but optionally using something like #7582.

[mattklein123]

Yeah +1 on contrib and we can generate 2 different images, one with contrib and one without? That should be really easy.

[htuch]

A couple of questions:

• How do we structure things API wise? Do we have a contrib API, or place the extensions in the official Envoy API from the get go? I think there is a balance between API stability and API review overhead here.
• Do we have differences in review policy in contrib? Pros are allowing faster experimentation and iteration (as you would have in a separate repo). The (very) big con is that this creates a very large step function to migration to core.

[ggreenway]

A couple of questions:

• How do we structure things API wise? Do we have a contrib API, or place the extensions in the official Envoy API from the get go? I think there is a balance between API stability and API review overhead here.
• Do we have differences in review policy in contrib? Pros are allowing faster experimentation and iteration (as you would have in a separate repo). The (very) big con is that this creates a very large step function to migration to core.

I'd say have contrib API in a separate directory, but in the same proto namespace so it can be moved from contrib to core without changing on-the-wire format, or really any source changes except maybe include paths.

[antoniovicente]

There has been the occasional issue of extensions that envoy core ends up having a tight dependency on, and build/test of Envoy core can't happen if the relevant extension is disabled and related code deleted. Having a separate directory for contrib and having an explicit goal of releasing with and without contrib sounds like a good step forward. Would it also make sense to have an envoy release without extensions? Granted, there are "extensions" like raw transport sockets and ssl transport sockets that Envoy does need in order to function.

[antoniovicente]

Regarding API and extensions: Would it make sense to define config protos for extensions in that extension's extension source directory? Nothing outside the extension needs to understand their config proto.

[lizan]

There has been the occasional issue of extensions that envoy core ends up having a tight dependency on, and build/test of Envoy core can't happen if the relevant extension is disabled and related code deleted. Having a separate directory for contrib and having an explicit goal of releasing with and without contrib sounds like a good step forward. Would it also make sense to have an envoy release without extensions? Granted, there are "extensions" like raw transport sockets and ssl transport sockets that Envoy does need in order to function.

We tightened the dependency enforcement and only a few extensions that Envoy core can depend on. I don't think the goal is to move all extensions to contrib, we need to define tiers of the extensions and decide which gets to the default image / additional flavored image, and supported by security team or not, etc.

[phlax]

seems like there is consensus not to move to a separate repo, so perhaps this point is moot...

the downside i see with that is that we would need to separate examples that required extensions, and i guess then replicate the ci for that

[htuch]

These are all good points. I've updated the title/top-level description to capture the contrib/ alternative.

[mattklein123]

I've put together a proposal for adding contrib/ to the Envoy repository. Please review! https://docs.google.com/document/d/1yl7GOZK1TDm_7vxQvt8UQEAu07UQFru1uEKXM6ZZg_g/edit#