feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow #2755

lichng · 2020-12-11T05:22:30Z

A Dataflow pipeline template that could import HBase data from an HBase snapshot into Cloud Bigtable.

bigtable-dataflow-parent/bigtable-beam-import/pom.xml

...main/java/com/google/cloud/bigtable/beam/hbasesnapshots/HBaseSnapshotInputConfiguration.java

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java

bigtable-dataflow-parent/bigtable-beam-import/src/main/resources/log4j.properties

...able-beam-import/src/test/java/com/google/cloud/bigtable/beam/hbasesnapshots/EndToEndIT.java

bigtable-dataflow-parent/bigtable-beam-import/pom.xml

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java

lichng · 2020-12-14T21:29:41Z

Hi @kolea2 @vermas2012 ,

This is good for another round of review. Please take another look.

We plan to keep tuning the performance, but it would be great if we can get this merge so we can start collaborating on the downstream processes(validation, syncTable).

bigtable-dataflow-parent/bigtable-beam-import/pom.xml

...-dataflow-parent/bigtable-beam-import/src/main/java/com/google/cloud/bigtable/beam/Main.java

...main/java/com/google/cloud/bigtable/beam/hbasesnapshots/HBaseSnapshotInputConfiguration.java

pom.xml

google-cla · 2020-12-29T17:08:18Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot · 2020-12-29T17:27:11Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs.

Serialization support

it to be more intuitive

1. use guava.version instead of beam-guava.version 2. fix typo

New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs.

1. use guava.version instead of beam-guava.version 2. fix typo

google-cla · 2020-12-29T17:36:04Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java

google-cla · 2021-01-21T14:47:01Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

Add cleanup phase

google-cla · 2021-01-22T21:41:25Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

filter out directories

google-cla · 2021-01-26T06:33:10Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2021-01-26T15:22:46Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2021-01-26T15:44:50Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2021-01-28T03:14:19Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2021-01-28T14:53:31Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

...n/java/com/google/cloud/bigtable/beam/hbasesnapshots/CleanupHBaseSnapshotRestoreFilesFn.java

...main/java/com/google/cloud/bigtable/beam/hbasesnapshots/HBaseSnapshotInputConfigBuilder.java

...n/java/com/google/cloud/bigtable/beam/hbasesnapshots/CleanupHBaseSnapshotRestoreFilesFn.java

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java

Minor code refactorig to reduce confusion

google-cla · 2021-02-01T18:39:44Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

google-cla · 2021-02-01T19:19:18Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

igorbernstein2

lgtm!

… Dataflow (googleapis#2755) * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Update document * Change the conf type for HBaseSnapshotInputConfiguration for Serialization support * Rename HBASE_ROOT_PATH to HBASE_EXPORT_ROOT_PATH in example doc for it to be more intuitive * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * Add the original Main.java under sequencefiles back * gcs connector still requires non-android guava version * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * switch HBasesnapshotConfiguration to a builder class * use DataflowRunner instead of DirectRunner for integration tests * revert pom file override * Remove all ValueProvider for now * Add gcsProject parameter and remove template related document * recover the dependency missed in the rebase * Update new files using latest header comment format and update year to 2021 * Remove workaround for BIGTABLE_BULK_AUTOFLUSH_MS_KEY * Exclude hbase-shaded-client * Clean up all transitive depdendencies on hbase-shaded-client * Add document for integration test generation instructions remove unnecessary code * Update document * Fail the pipeline building when there is an exception configuring input updated unit test more comments * renaming according to review comments * update comments * More document about hbase snapshot file structure * System.out -> LOG * throw out exception instead of terminating JVM * Remove outside visible parameter restoreDir, use a default dir instead Add cleanup phase * use pattern without ending '/' * use listObject instead of match since GcsUtil expand intentionally filter out directories * Using a unique suffix for restore dir to avoid conflict * Add dependency to pom.xml * minimize accessibility for class * Fix typo and Add header comment for CleanupHBaseSnapshotRestoreFilesFn * Adding more error messages for HBaseSnapshotInputConfigBuilder Minor code refactorig to reduce confusion * Add document about how to handle temp files during job failures

… Dataflow (googleapis#2755) * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Update document * Change the conf type for HBaseSnapshotInputConfiguration for Serialization support * Rename HBASE_ROOT_PATH to HBASE_EXPORT_ROOT_PATH in example doc for it to be more intuitive * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * Add the original Main.java under sequencefiles back * gcs connector still requires non-android guava version * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * switch HBasesnapshotConfiguration to a builder class * use DataflowRunner instead of DirectRunner for integration tests * revert pom file override * Remove all ValueProvider for now * Add gcsProject parameter and remove template related document * recover the dependency missed in the rebase * Update new files using latest header comment format and update year to 2021 * Remove workaround for BIGTABLE_BULK_AUTOFLUSH_MS_KEY * Exclude hbase-shaded-client * Clean up all transitive depdendencies on hbase-shaded-client * Add document for integration test generation instructions remove unnecessary code * Update document * Fail the pipeline building when there is an exception configuring input updated unit test more comments * renaming according to review comments * update comments * More document about hbase snapshot file structure * System.out -> LOG * throw out exception instead of terminating JVM * Remove outside visible parameter restoreDir, use a default dir instead Add cleanup phase * use pattern without ending '/' * use listObject instead of match since GcsUtil expand intentionally filter out directories * Using a unique suffix for restore dir to avoid conflict * Add dependency to pom.xml * minimize accessibility for class * Fix typo and Add header comment for CleanupHBaseSnapshotRestoreFilesFn * Adding more error messages for HBaseSnapshotInputConfigBuilder Minor code refactorig to reduce confusion * Add document about how to handle temp files during job failures (cherry picked from commit 5b3ab2b)

…rt validation (#2958) * feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow (#2755) * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Update document * Change the conf type for HBaseSnapshotInputConfiguration for Serialization support * Rename HBASE_ROOT_PATH to HBASE_EXPORT_ROOT_PATH in example doc for it to be more intuitive * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * Add the original Main.java under sequencefiles back * gcs connector still requires non-android guava version * Support import from HBase snapshot New files for configuring HBaseSnapshotInputFormat resovle version conflict and upgrade Beam version to 2.24.0 revert disk option change, not enough quota Code reorg code reduction Refactor naming Add integration config Add unit test for HBaseSnapshotInputConfiguration Set up skeleton for integration testing Ship test data with code, integration tests pass Clean up code for PR Add HBase commands that generates our test snapshot Addressing review comments 1. revert pom file overrides for SkipITs 2. Store SerializableConfiguration as member variable 3. Rever log4j.properties 4. Disable BIGTABLE_BULK_AUTOFLUSH_MS_KEY to prevent bulk mutation failures failing the jobs. * Addressing the review comments: 1. use guava.version instead of beam-guava.version 2. fix typo * switch HBasesnapshotConfiguration to a builder class * use DataflowRunner instead of DirectRunner for integration tests * revert pom file override * Remove all ValueProvider for now * Add gcsProject parameter and remove template related document * recover the dependency missed in the rebase * Update new files using latest header comment format and update year to 2021 * Remove workaround for BIGTABLE_BULK_AUTOFLUSH_MS_KEY * Exclude hbase-shaded-client * Clean up all transitive depdendencies on hbase-shaded-client * Add document for integration test generation instructions remove unnecessary code * Update document * Fail the pipeline building when there is an exception configuring input updated unit test more comments * renaming according to review comments * update comments * More document about hbase snapshot file structure * System.out -> LOG * throw out exception instead of terminating JVM * Remove outside visible parameter restoreDir, use a default dir instead Add cleanup phase * use pattern without ending '/' * use listObject instead of match since GcsUtil expand intentionally filter out directories * Using a unique suffix for restore dir to avoid conflict * Add dependency to pom.xml * minimize accessibility for class * Fix typo and Add header comment for CleanupHBaseSnapshotRestoreFilesFn * Adding more error messages for HBaseSnapshotInputConfigBuilder Minor code refactorig to reduce confusion * Add document about how to handle temp files during job failures (cherry picked from commit 5b3ab2b) * feat: Add a new pipeline to validate data imported from HBase (#2828) * feat: add a new pipeline to validate data imported into cloud bigtable from HBase. (cherry picked from commit fa07a90)

product-auto-label bot added the api: bigtable Issues related to the googleapis/java-bigtable-hbase API. label Dec 11, 2020

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 11, 2020

lichng changed the title ~~Ability to import HBase Snapshot into Cloud Bigtable using Dataflow~~ feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow Dec 11, 2020

lichng force-pushed the snapshot-import branch from df8bf6f to b99c98a Compare December 11, 2020 15:53

lichng marked this pull request as ready for review December 11, 2020 17:10

lichng requested review from a team as code owners December 11, 2020 17:10

vermas2012 reviewed Dec 14, 2020

View reviewed changes

kolea2 reviewed Dec 14, 2020

View reviewed changes

bigtable-dataflow-parent/bigtable-beam-import/pom.xml Outdated Show resolved Hide resolved

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java Outdated Show resolved Hide resolved

lichng force-pushed the snapshot-import branch from 47e90d7 to ed26647 Compare December 14, 2020 20:16

igorbernstein2 reviewed Dec 17, 2020

View reviewed changes

lichng force-pushed the snapshot-import branch from da773b3 to bac51a1 Compare December 22, 2020 21:11

google-cla bot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Dec 29, 2020

lichng added 9 commits December 29, 2020 17:32

Update document

7d63490

Change the conf type for HBaseSnapshotInputConfiguration for

6c0ed5e

Serialization support

Rename HBASE_ROOT_PATH to HBASE_EXPORT_ROOT_PATH in example doc for

fc728a8

it to be more intuitive

Addressing the review comments:

316c0aa

1. use guava.version instead of beam-guava.version 2. fix typo

Add the original Main.java under sequencefiles back

6188efa

gcs connector still requires non-android guava version

f621dc0

Addressing the review comments:

53f73bc

1. use guava.version instead of beam-guava.version 2. fix typo

lichng force-pushed the snapshot-import branch from 2520346 to 53f73bc Compare December 29, 2020 17:35

google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Dec 29, 2020

igorbernstein2 reviewed Jan 20, 2021

View reviewed changes

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java Outdated Show resolved Hide resolved

igorbernstein2 reviewed Jan 20, 2021

View reviewed changes

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java Outdated Show resolved Hide resolved

igorbernstein2 reviewed Jan 20, 2021

View reviewed changes

.../src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java Outdated Show resolved Hide resolved

throw out exception instead of terminating JVM

ce4e2cf

lichng requested a review from igorbernstein2 January 21, 2021 17:47

Remove outside visible parameter restoreDir, use a default dir instead

fc6e1e4

Add cleanup phase

lichng added 2 commits January 24, 2021 04:13

use pattern without ending '/'

e871418

use listObject instead of match since GcsUtil expand intentionally

6f15d81

filter out directories

Using a unique suffix for restore dir to avoid conflict

5db7ce6

Add dependency to pom.xml

87b0eff

minimize accessibility for class

c967d17

Fix typo and Add header comment for CleanupHBaseSnapshotRestoreFilesFn

59e7a55

igorbernstein2 requested changes Feb 1, 2021

View reviewed changes

Adding more error messages for HBaseSnapshotInputConfigBuilder

2596d88

Minor code refactorig to reduce confusion

Add document about how to handle temp files during job failures

94270af

igorbernstein2 approved these changes Feb 1, 2021

View reviewed changes

igorbernstein2 merged commit 5b3ab2b into googleapis:master Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow #2755

feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow #2755

lichng commented Dec 11, 2020 •

edited

Loading

lichng commented Dec 14, 2020

google-cla bot commented Dec 29, 2020

googlebot commented Dec 29, 2020

google-cla bot commented Dec 29, 2020

google-cla bot commented Jan 21, 2021

google-cla bot commented Jan 22, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 28, 2021

google-cla bot commented Jan 28, 2021

google-cla bot commented Feb 1, 2021

google-cla bot commented Feb 1, 2021

igorbernstein2 left a comment

feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow #2755

feat: Ability to import HBase Snapshot data into Cloud Bigtable using Dataflow #2755

Conversation

lichng commented Dec 11, 2020 • edited Loading

lichng commented Dec 14, 2020

google-cla bot commented Dec 29, 2020

googlebot commented Dec 29, 2020

google-cla bot commented Dec 29, 2020

google-cla bot commented Jan 21, 2021

google-cla bot commented Jan 22, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 26, 2021

google-cla bot commented Jan 28, 2021

google-cla bot commented Jan 28, 2021

google-cla bot commented Feb 1, 2021

google-cla bot commented Feb 1, 2021

igorbernstein2 left a comment

Choose a reason for hiding this comment

lichng commented Dec 11, 2020 •

edited

Loading