feat: move new client script #2333

JoeWang1127 · 2024-01-03T15:30:56Z

In this PR:

Migrate new-client.py from google-cloud-java, because we will be using new-client.py to other use cases than google-cloud-java repository, such as self-service client library generation.
Separate repo-level process script to library_generation/repo-level-postprocess

To verify new-client.py:

pull the branch.
pull the main branch of google-cloud-java.
copy library_generation folder to root directory of google-cloud-java.
in google-cloud-java, remove a newly added module, e.g., java-cloudquotas (PR).
regenerate the module:

python library_generation/new_client/new-client.py generate --api_shortname=cloudquotas --proto-path=google/api/cloudquotas --name-pretty="Cloud Quotas API" --product-docs="https://cloud.google.com/cloudquotas/docs/" --api-description="Cloud Quotas API provides GCP service consumers with management and
    observability for resource usage, quotas, and restrictions of the services
    they consume."

use git status and git diff to verify the difference.

This is part of milestone 2 of hermetic build project plan.

Note that I didn't test the script against split repositories because the main use case of this script is to generate a module in google-cloud-java.

…dk-platform-java into feat/move-new-client-script

JoeWang1127 · 2024-01-23T17:50:17Z

I don't like release managers need to specify this value. Is there a way not to specify this argument?

One option is add a release-please annotation so that the version num is the same as the latest release version.

@suztomo what do you think this idea.

blakeli0 · 2024-01-23T17:59:11Z

I don't like release managers need to specify this value. Is there a way not to specify this argument?

One option is add a release-please annotation so that the version num is the same as the latest release version.

@suztomo what do you think this idea.

Also I want to note that this is a temporary solution, we will add a new section for a new client library in the generation_config.yaml in the future. So specifying a generator version looks fine to me, and I would prefer us not to optimize too much for a temporary solution if it takes more than like 30 minutes.

suztomo · 2024-01-23T18:06:01Z

I agree that Release Please is overkill for this work. Here is the shell script to get the version:

# Temporary solution to get gapic-generator-java version until the configuration file is ready
$ V=$(curl --silent 'https://raw.githubusercontent.com/googleapis/googleapis/master/WORKSPACE' |perl -nle 'print $1 if m/_gapic_generator_java_version\s+=\s+\"(.+)\"/')
$ echo $V
2.32.0

JoeWang1127 · 2024-01-23T18:09:22Z

I agree that Release Please is overkill for this work. Here is the shell script to get the version:

# Temporary solution to get gapic-generator-java version until the configuration file is ready
$ V=$(curl --silent 'https://raw.githubusercontent.com/googleapis/googleapis/master/WORKSPACE' |perl -nle 'print $1 if m/_gapic_generator_java_version\s+=\s+\"(.+)\"/')
$ echo $V
2.32.0

Thanks for the help.

JoeWang1127 · 2024-01-23T22:14:16Z

library_generation/new_client/new-client.py

+        versioned_path=versioned_proto_path,
+    )
+    repo_root_dir = Path(f"{sys.path[0]}/../../").resolve()
+    generator_version = subprocess.check_output(


The version of generator is now parsed from WORKSPACE. This is a temporary fix so that release manager doesn't need to specify this value.

The plan is to use the version in config file later.

JoeWang1127 · 2024-01-24T00:15:42Z

library_generation/new_client/templates/owlbot.py.j2

+
+s.remove_staging_dirs()
+{% if should_include_templates %}java.common_templates(monorepo=True, {% if template_excludes %}excludes=[
+{%- for exclude in template_excludes %}


Change template so that update_owlbot_postprocessor_config.sh is not needed.

JoeWang1127 · 2024-01-24T02:16:24Z

library_generation/owlbot/src/fix-poms.py

        templates.render(
            template_name="bom_pom.xml.j2",
            output_name=f"{artifact_id}-bom/pom.xml",
            repo=repo_metadata["repo"],
            name=name,
            modules=modules,
            main_module=main_module,
+            monorepo=monorepo,
+            monorepo_version=monorepo_version


Change the template so that we can remove set_parent_pom.sh.

JoeWang1127 · 2024-01-24T20:16:02Z

library_generation/owlbot/src/fix-poms.py

@@ -386,11 +399,12 @@ def main(versions_file, monorepo):
                module=required_dependencies[path],
                parent_module=parent_module,
                main_module=main_module,
+                monorepo=monorepo,


Change the proto pom template to remove consolidate_config.sh.

suztomo

Looking forward to see the observations that it works for shopping CSS.

JoeWang1127 · 2024-01-24T21:19:17Z

Steps to regenerate shopping CSS:

Checkout main branch of google-cloud-java and pull the latest change.
Copy library_generation to google-cloud-java's root directory.
Install python package:

python -m pip -r install library_generation/new_client/requirements.in

Install google-cloud-java-jar-parent and google-cloud-bom-parent if they are SNAPSHOT version, since mvn fmt:format requires all parent poms are resolved.
Remove shopping CSS module in google-cloud-java:

rm -rf java-shopping-css

Run the generation command:

python library_generation/new_client/new-client.py generate --api_shortname="css" --proto-path="google/shopping/css" --name-pretty="CSS API" --product-docs="https://developers.google.com/comparison-shopping-services/api" --api-description="The CSS API is used to manage your CSS and control your CSS Products portfolio" --distribution-name="com.google.shopping:google-shopping-css" --destination-name=shopping-css

Verify the difference: log.
The difference of clirr-ignored-differences.xml should be fixed after cl/600749234 is propogated to google-cloud-java.

sonarqubecloud · 2024-01-25T01:07:04Z

Quality Gate passed for 'gapic-generator-java-root'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarqubecloud · 2024-01-25T01:10:55Z

Quality Gate passed for 'java_showcase_integration_tests'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

suztomo · 2024-01-25T13:54:13Z

Verify the difference: log.

That looks good to me. Thank you.

blakeli0 · 2024-01-25T16:11:59Z

library_generation/owlbot/src/fix-poms.py

@@ -477,52 +493,57 @@ def main(versions_file, monorepo):
            update_bom_pom(f"{artifact_id}-bom/pom.xml", modules)
    elif artifact_id+"-bom" not in excluded_poms_list:
        print("creating missing bom pom.xml")
+        monorepo_version = __get_monorepo_version(versions_file) \


I may missed this in earlier reviews, can you please remind me why we need to get monorepo_version now?

I want to generate the pom.xml in *-bom without using a repo-level post processing script (set_parent_pom.sh in this case) and run mvn fmt:format after the pom generation.

This requires to know the current version of gapic-libraries-bom and google-cloud-jar-parent, which is the same as the monorepo version.

JoeWang1127 added 2 commits January 2, 2024 19:06

feat: add new-client.py

df960de

refactor code

61f855f

product-auto-label bot added the size: l Pull request size is large. label Jan 3, 2024

JoeWang1127 and others added 7 commits January 3, 2024 11:23

clone googleapis

1a98967

Merge branch 'main' into feat/move-new-client-script

f641afc

add build parser

d16e583

parse inputs from BUILD

d44e1e6

find versioned dir within proto_path

f62e9ec

fix postprocessing, use google-java-format

a6b129d

migirate shell scripts

12bc62f

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Jan 5, 2024

JoeWang1127 added 18 commits January 5, 2024 14:52

change depth

d02507b

add generate_gapic_bom.sh

01a5bcb

process root pom

e2b2e59

only generating gapic-libraries-bom if it already exists

81fb67a

add consolidate_config.sh

78a2259

add generation dir

32a2c95

use variable for script dir

9c6b987

add set_parent_pom.sh

b25abc0

add apply_current_versions.sh

656d771

add click option for versions.txt

a3d91eb

change dir

90a56cd

change find command

7408d42

add readme_update.sh

bf5bc08

Merge branch 'feat/move-new-client-script' of github.com:googleapis/s…

cb93c7b

…dk-platform-java into feat/move-new-client-script

change script path

68a51ac

refactor

bac6008

code refactor

4fd8a17

add generator version

828e5bf

parse generator version from WORKSPACE

46a6b32

JoeWang1127 commented Jan 23, 2024

View reviewed changes

JoeWang1127 added 2 commits January 23, 2024 22:22

remove readme_update.sh

bda8041

remove update_owlbot_postprocessor_config.sh

4f3fccd

JoeWang1127 commented Jan 24, 2024

View reviewed changes

JoeWang1127 added 2 commits January 24, 2024 00:21

remove delete_non_generated_samples.sh

b33c765

remove set_parent_pom.sh

594177a

JoeWang1127 commented Jan 24, 2024

View reviewed changes

JoeWang1127 added 5 commits January 24, 2024 14:21

Merge branch 'main' into feat/move-new-client-script

665316c

add destination_name

4874bbd

restore format plugin

1d0ca6c

remove gitignore

68c3eab

remove consolidate_config.sh

f9b9ad5

JoeWang1127 commented Jan 24, 2024

View reviewed changes

suztomo approved these changes Jan 24, 2024

View reviewed changes

JoeWang1127 added 2 commits January 25, 2024 00:42

Merge branch 'main' into feat/move-new-client-script

45f20b3

change output directory

787a7a1

blakeli0 reviewed Jan 25, 2024

View reviewed changes

blakeli0 approved these changes Jan 25, 2024

View reviewed changes

JoeWang1127 merged commit acdde47 into main Jan 25, 2024
46 checks passed

JoeWang1127 deleted the feat/move-new-client-script branch January 25, 2024 16:31

release-please bot mentioned this pull request Jan 25, 2024

chore(main): release 2.34.0 #2417

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: move new client script #2333

feat: move new client script #2333

JoeWang1127 commented Jan 3, 2024 •

edited

Loading

JoeWang1127 commented Jan 23, 2024

blakeli0 commented Jan 23, 2024

suztomo commented Jan 23, 2024 •

edited

Loading

JoeWang1127 commented Jan 23, 2024

JoeWang1127 Jan 23, 2024

JoeWang1127 Jan 24, 2024

JoeWang1127 Jan 24, 2024

JoeWang1127 Jan 24, 2024

suztomo left a comment

JoeWang1127 commented Jan 24, 2024 •

edited

Loading

sonarqubecloud bot commented Jan 25, 2024

sonarqubecloud bot commented Jan 25, 2024

suztomo commented Jan 25, 2024

blakeli0 Jan 25, 2024

JoeWang1127 Jan 25, 2024

feat: move new client script #2333

feat: move new client script #2333

Conversation

JoeWang1127 commented Jan 3, 2024 • edited Loading

JoeWang1127 commented Jan 23, 2024

blakeli0 commented Jan 23, 2024

suztomo commented Jan 23, 2024 • edited Loading

JoeWang1127 commented Jan 23, 2024

JoeWang1127 Jan 23, 2024

Choose a reason for hiding this comment

JoeWang1127 Jan 24, 2024

Choose a reason for hiding this comment

JoeWang1127 Jan 24, 2024

Choose a reason for hiding this comment

JoeWang1127 Jan 24, 2024

Choose a reason for hiding this comment

suztomo left a comment

Choose a reason for hiding this comment

JoeWang1127 commented Jan 24, 2024 • edited Loading

sonarqubecloud bot commented Jan 25, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarqubecloud bot commented Jan 25, 2024

Quality Gate passed for 'java_showcase_integration_tests'

suztomo commented Jan 25, 2024

blakeli0 Jan 25, 2024

Choose a reason for hiding this comment

JoeWang1127 Jan 25, 2024

Choose a reason for hiding this comment

JoeWang1127 commented Jan 3, 2024 •

edited

Loading

suztomo commented Jan 23, 2024 •

edited

Loading

JoeWang1127 commented Jan 24, 2024 •

edited

Loading