fix for adding unexpected Empty Records in Nested Arrays in BigQueryIO

[stankiewicz]

fixes #33842

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.
R: @damondouglas for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[stankiewicz]

@ahmedabu98 can you take a look at this? thanks!

[stankiewicz]

do not merge yet, I think I found another bug.

[ahmedabu98]

This LGTM! Thanks for taking this @stankiewicz

Lmk when it's ready to merge

[stankiewicz]

issue I see additional problem here: StorageApiWriteUnshardedRecords

If we have a row with known fields (name:string, surname:string) and one unknown (foo:string) , this line will concat and not cause schema issues.

but if unknown field is part of repeated structure,
e.g. (name:string, phone:(repeated struct type:string, num:string) ) and we have unknown in repeated struct (phone: [ null, {favourite:yes} ]) concat doesn't work well for some reason.

For current bug, it is resolved - it won't add empty records so we can merge it.

Added issue for concatenating unknown nested fields which may require bigger changes #34145