Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: re-enable HW processing for hermetic build scripts #2505

Merged
merged 40 commits into from
Mar 4, 2024

Conversation

diegomarquezp
Copy link
Contributor

@diegomarquezp diegomarquezp commented Feb 27, 2024

re-enables the changes from #2342 as part of milestone 3.2 of Hermetic Build

Changes:

  • add support for HW libs in integration_tests.py
  • adapt configuration of docker volumes
  • adapt usage of fix-poms.py to use monorepo and split_repo modes
  • fix temp_destination_path in generate_composed_library.py in order to avoid collisions
    • For example google/bigtable/v2 and google/bigtable/admin/v2 would both be sent to a temporal java-bigtable-v2 folder, spoiling the files being transferred via copy-code.
    • The fix is to have a proto_path-based folder name to achieve uniqueness

After this PR:

  • Reconfigure the nightly CI in google-cloud-java to use the updated volume mapping

@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label Feb 27, 2024
@diegomarquezp diegomarquezp changed the title chore: [wip] re-enable HW processing for hermetic build scripts chore: re-enable HW processing for hermetic build scripts Feb 27, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Feb 28, 2024
@diegomarquezp
Copy link
Contributor Author

Can you link the corresponding task in project plan in PR description?

Done

"repo_short": "java-bare-metal-solution",
"distribution_name": "com.google.cloud:google-cloud-bare-metal-solution",
"api_id": "baremetalsolution.googleapis.com",
"library_type": "GAPIC_AUTO",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to change this value to GAPIC_COMBO?

Copy link
Contributor Author

@diegomarquezp diegomarquezp Feb 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: Just hardcode it in the test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -102,6 +108,11 @@ def generate_from_yaml(
versions_file=repo_config.versions_file,
)

# we skip repo_level_postprocess if not in a monorepo since it's meant for
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it was not introduced in this PR, can we rename repo_level_postprocess to monorepo_postprocess if it's meant for monorepo only?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Renamed to monorepo_postprocessing

@@ -46,6 +46,7 @@ def __init__(
rpc_documentation: Optional[str] = None,
cloud_api: Optional[bool] = True,
requires_billing: Optional[bool] = True,
extra_versioned_modules: Optional[str] = None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diegomarquezp Going forward, I'm not sure accommodating every non-standard field in repo-metadata.json in our scripts is the way to go, we may want to leave the responsibility to each handwritten library owner. For now, can you please document this new parameter in go/java-hermetic-build-config-spec?

@@ -14,5 +14,6 @@
"library_type": "GAPIC_AUTO",
"requires_billing": true,
"rest_documentation": "https://cloud.google.com/bare-metal/docs/reference/rest",
"rpc_documentation": "https://cloud.google.com/bare-metal/docs/reference/rpc"
"rpc_documentation": "https://cloud.google.com/bare-metal/docs/reference/rpc",
"extra_versioned_modules": "test-module"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This parameter should not be in monorepo goldens. Do google-cloud-java and java-bigtable share the same test setup?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I treated this repo-metadata entry as monorepo-agnostic, but I agree it's closer to real life to not include it here. I removed it and adapted the test accordingly.

Copy link

Quality Gate Passed Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

Copy link

Quality Gate Passed Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

@diegomarquezp diegomarquezp merged commit 46c08a2 into main Mar 4, 2024
45 checks passed
@diegomarquezp diegomarquezp deleted the restore-hermetic-build-for-hw-libs branch March 4, 2024 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants