Bump grpc from 1.26.0 to 1.31.1 #12226

dmivankov · 2020-10-07T11:33:46Z

Fixes having external dependencies without checksum

boringssl (each download was timestamped, but otherwise stable)
bazel_skylark was overriden to be master (sic!)

There doesn't seem to be many breaking/big changes up to grpc 1.31.1

removal of xds-experimental URI scheme
removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL
enable TLS 1.3 in the C-core and all wrapped languages
some of bazel-related patches got merged in
https://github.com/grpc/grpc/releases

How to check whether certain dependency has a checksum
bazel query //external:bazel_skylib --output build
bazel query //external:boringssl --output build

How to find (almost?) all problematic dependencies
compare output of
bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"'
vs
bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"'

Note that it looks for string sha256 and misses dict sha256 for
distdir_tar rules - those are false positive currently.

@name

Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently.

Note: didn't older version have to be declared too?

meteorcloudy · 2020-10-08T08:50:31Z

Nice, thanks for the update! This PR looks good to me.

But we cannot import a PR with changes both inside and outside of the third_party directory, could you please split it into several PRs? We have to do it in a way that won't break Bazel, an example of changes is:
#11807
#12108
#12139

dmivankov · 2020-10-08T08:55:48Z

Thanks, will look into that. Do you have more details on what could break with a single PR? As I see there are bootstrapping and other tests in CI

meteorcloudy · 2020-10-08T09:00:23Z

Yeah, basically I meant the presubmit tests should pass. If it's green, we are fine.

dmivankov · 2020-10-08T09:05:18Z

I will probably have to add grpc version suffix to
third_party/grpc/BUILD
third_party/grpc/bazel.patch
third_party/grpc/bazel/*
third_party/grpc/compiler/*
otherwise can't simply add new grpc version alongside previous one

dmivankov · 2020-10-08T09:09:54Z

Ah, nice, so all is good with the current state of PR? :)

meteorcloudy · 2020-10-08T09:56:37Z

I will probably have to add grpc version suffix to
third_party/grpc/BUILD
third_party/grpc/bazel.patch
third_party/grpc/bazel/*
third_party/grpc/compiler/*
otherwise can't simply add new grpc version alongside previous one

I think the grpc java compiler version doesn't have to exactly match the grpc cpp version. So you probably don't have to do that. It's just the grpc_1.26.0.patch file has to match the version defined in WORKSPACE.
You can do this

First add the new patch file grpc_1.31.1.patch and upgrade everything under third_party (bazel/ , compiler/ , grpc jars)
Upgrade the grpc version in WORKSPACE to 1.31.1
Remove old patch files.

… from

… it came from" Actually, embedded_tools_src patch comes from 22d7e77: Make it possible to do `bazel query @bazel_tools//...` without errors. It just patched third_party/grpc/BUILD without updating third_party/grpc/bazel.patch Now patch is made up to date. Note: it could be that PR check doesn't test the `bazel query @bazel_tools//...` case This reverts commit 3f11beb.

@name

PART 1: prepare third_party/grpc files for new version Composed PR: bazelbuild#12226 Note: generate_cc.bzl and protobuf.bzl are modified in place and already affect the build. But the change seems to be harmless (adding explicit ProtoInfo load from @rules_proto). Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently.

@name

PART 2: switch to grpc 1.31.1 Composed PR: bazelbuild#12226 Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently.

@name

PART 3: remove grpc 1.26.0 from third_party/grpc Composed PR: bazelbuild#12226 Note: bootstrap jars & java_plugin remain to be 1.26.0 though Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently.

dmivankov · 2020-10-09T07:21:27Z

keeping bootstrap jars and protoc java plugin at 1.26.0
split into 3: [1/3] Bump grpc from 1.26.0 to 1.31.1 #12235 (add grpc 1.31.1 patches), [2/3] Bump grpc from 1.26.0 to 1.31.1 #12236 (switch to 1.31.1) and [3/3] Bump grpc from 1.26.0 to 1.31.1 #12237 (drop 1.26.0 patches)
keeping this one open for now for full diff overview, PR checks & comments

meteorcloudy · 2020-10-09T08:17:55Z

Thanks, do you plan to upgrade the jars later?

@name

PART 1: prepare third_party/grpc files for new version Composed PR: #12226 Note: generate_cc.bzl and protobuf.bzl are modified in place and already affect the build. But the change seems to be harmless (adding explicit ProtoInfo load from @rules_proto). Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes #12235

dmivankov · 2020-10-09T08:24:55Z

Bootstrap jars seem to have a good amount of bugfixes, so would be good to bump them https://github.com/grpc/grpc-java/releases
But actually there seem to be netty dependency bump there - would need to double-check version and make it a separate PR

@name

PART 2: switch to grpc 1.31.1 Composed PR: bazelbuild#12226 Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently.

meteorcloudy · 2020-10-09T08:27:38Z

Sounds great!

@name

PART 3: remove grpc 1.26.0 from third_party/grpc Composed PR: #12226 Note: bootstrap jars & java_plugin remain to be 1.26.0 though Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes #12237

meteorcloudy · 2020-10-09T13:21:39Z

split into 3: #12235 (add grpc 1.31.1 patches), #12236 (switch to 1.31.1) and #12237 (drop 1.26.0 patches)

All three PRs have been merged now.

@name

PART 1: prepare third_party/grpc files for new version Composed PR: bazelbuild#12226 Note: generate_cc.bzl and protobuf.bzl are modified in place and already affect the build. But the change seems to be harmless (adding explicit ProtoInfo load from @rules_proto). Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes bazelbuild#12235

@name

PART 3: remove grpc 1.26.0 from third_party/grpc Composed PR: bazelbuild#12226 Note: bootstrap jars & java_plugin remain to be 1.26.0 though Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes bazelbuild#12237

@name

PART 1: prepare third_party/grpc files for new version Composed PR: bazelbuild#12226 Note: generate_cc.bzl and protobuf.bzl are modified in place and already affect the build. But the change seems to be harmless (adding explicit ProtoInfo load from @rules_proto). Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes bazelbuild#12235

@name

PART 3: remove grpc 1.26.0 from third_party/grpc Composed PR: bazelbuild#12226 Note: bootstrap jars & java_plugin remain to be 1.26.0 though Fixes having external dependencies without checksum - boringssl (each download was timestamped, but otherwise stable) - bazel_skylark was overriden to be master (sic!) There doesn't seem to be many breaking/big changes up to grpc 1.31.1 - removal of xds-experimental URI scheme - removal of MAX_EPOLL_EVENTS_HANDLED_EACH_POLL_CALL - enable TLS 1.3 in the C-core and all wrapped languages - some of bazel-related patches got merged in https://github.com/grpc/grpc/releases How to check whether certain dependency has a checksum bazel query //external:bazel_skylib --output build bazel query //external:boringssl --output build How to find (almost?) all problematic dependencies compare output of bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | ."@name"' vs bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all)' --output xml | xq '.query.rule[] | select (.string[]."@name" | contains("sha256")) | ."@name"' Note that it looks for string sha256 and misses dict sha256 for distdir_tar rules - those are false positive currently. Closes bazelbuild#12237

google-cla bot added the cla: yes label Oct 7, 2020

dmivankov mentioned this pull request Oct 7, 2020

bazel: update to 3.5.0 NixOS/nixpkgs#97497

Closed

dmivankov added 4 commits October 7, 2020 15:34

fix grpc jar version bootstrap

6cacf6c

update hardcoded upb commit&hash

729c599

declare new google/re2 dependency from newer grpc

bb3f895

declare abseil/abseil-cpp dependency

c514b1b

Note: didn't older version have to be declared too?

meisterT assigned meteorcloudy Oct 7, 2020

meteorcloudy self-requested a review October 8, 2020 08:42

dmivankov added 4 commits October 8, 2020 13:51

let's try keeping java plugin and bootstrap jars at 1.26.0

041442f

remove embedded_tools_srcs patch, don't really remember where it came…

3f11beb

… from

remove accidental tab

b67786e

dmivankov mentioned this pull request Oct 9, 2020

[1/3] Bump grpc from 1.26.0 to 1.31.1 #12235

Closed

meteorcloudy closed this Oct 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump grpc from 1.26.0 to 1.31.1 #12226

Bump grpc from 1.26.0 to 1.31.1 #12226

dmivankov commented Oct 7, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 8, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 8, 2020 •

edited

Loading

dmivankov commented Oct 8, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 9, 2020 •

edited

Loading

meteorcloudy commented Oct 9, 2020

dmivankov commented Oct 9, 2020

meteorcloudy commented Oct 9, 2020

meteorcloudy commented Oct 9, 2020

Bump grpc from 1.26.0 to 1.31.1 #12226

Bump grpc from 1.26.0 to 1.31.1 #12226

Conversation

dmivankov commented Oct 7, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 8, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 8, 2020 • edited Loading

dmivankov commented Oct 8, 2020

meteorcloudy commented Oct 8, 2020

dmivankov commented Oct 9, 2020 • edited Loading

meteorcloudy commented Oct 9, 2020

dmivankov commented Oct 9, 2020

meteorcloudy commented Oct 9, 2020

meteorcloudy commented Oct 9, 2020

dmivankov commented Oct 8, 2020 •

edited

Loading

dmivankov commented Oct 9, 2020 •

edited

Loading