Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add PG OID support #2736

Merged
merged 53 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
6b3f572
feat: add PG OID support
tlhquynh Nov 20, 2023
929abf4
chore: fix lint errors
tlhquynh Nov 20, 2023
4e7ea9d
Resolved merge conflict from branch 'main' into pg-oid.
tlhquynh Dec 29, 2023
0489d9a
Merge branch 'main' into pg-oid
tlhquynh Mar 14, 2024
e5a39c5
Update PG.OID implementation according to recent changes.
tlhquynh Mar 18, 2024
537de96
Update PG.OID implementation according to recent changes.
tlhquynh Mar 18, 2024
f9ea807
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 22, 2024
4cb354a
chore: keep session pool ordering when pinging (#2695)
olavloite Mar 15, 2024
e505b9e
deps: update dependency com.google.cloud:google-cloud-monitoring to v…
renovate-bot Mar 18, 2024
0184554
feat: allow attempt direct path xds via env var (#2950)
HailongWen Mar 19, 2024
1d756a3
build(deps): update dependency org.apache.maven.plugins:maven-compile…
renovate-bot Mar 19, 2024
f6193f9
build(deps): update dependency org.apache.maven.plugins:maven-assembl…
renovate-bot Mar 19, 2024
aed1223
deps: update dependency com.google.cloud:sdk-platform-java-config to …
renovate-bot Mar 19, 2024
bf2361b
refactor: move skip methods to abstract parser (#2948)
olavloite Mar 19, 2024
57edce9
fix: return type of max commit delay option. (#2953)
arpan14 Mar 19, 2024
d7ba42b
refactor: generalize skip methods (#2949)
olavloite Mar 19, 2024
50e2887
perf: keep comments when searching for params (#2951)
olavloite Mar 19, 2024
6258293
chore: randomize session pool order based on TPS (#2792)
olavloite Mar 19, 2024
d94922f
chore(main): release 6.62.0 (#2940)
release-please[bot] Mar 19, 2024
bd82e0c
chore(main): release 6.62.1-SNAPSHOT (#2957)
release-please[bot] Mar 19, 2024
a0a2e8d
chore(deps): update dependency com.google.cloud:google-cloud-spanner …
renovate-bot Mar 20, 2024
eaa05da
chore: add session pool options for multiplexed session. (#2960)
arpan14 Mar 22, 2024
8e4ee03
deps: update dependency com.google.cloud:google-cloud-trace to v2.38.…
renovate-bot Mar 24, 2024
ebf8723
chore: add new members in SessionImpl for multiplexed session. Add a …
arpan14 Mar 24, 2024
358e8d3
Update .gitignore to remove IDE specific files and remove unnecessary…
tlhquynh Mar 25, 2024
1e99310
Remove PG.OID external getters.
tlhquynh Mar 25, 2024
039079e
chore: generalise session pool class for multiplexed session. (#2964)
arpan14 Mar 25, 2024
b21e262
Merge branch 'main' into pg-oid
tlhquynh Mar 26, 2024
6f971a7
Merge branch 'main' into pg-oid
tlhquynh Mar 28, 2024
5df12ca
chore: emove unnecessary debug.
tlhquynh Mar 28, 2024
4b12761
chore: add multiplexed session implementations for CachedSession/Sess…
arpan14 Mar 28, 2024
60712bd
Remove internal PG.OID getters.
tlhquynh Apr 3, 2024
19fbdfb
deps: update dependency com.google.cloud:google-cloud-monitoring to v…
renovate-bot Mar 28, 2024
f3f6113
chore(main): release 6.62.1 (#2968)
release-please[bot] Mar 28, 2024
8ac2a16
chore(main): release 6.62.2-SNAPSHOT (#2983)
release-please[bot] Mar 28, 2024
231273e
feat: add support for transaction-level exclusion from change streams…
dengwe1 Mar 28, 2024
80ade2e
deps: update dependency com.google.cloud:google-cloud-trace to v2.39.…
renovate-bot Mar 29, 2024
ecb87e7
deps: update dependency commons-io:commons-io to v2.16.0 (#2986)
renovate-bot Mar 29, 2024
7444b29
deps: update dependency com.google.cloud:google-cloud-monitoring to v…
renovate-bot Mar 30, 2024
8a9584b
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Mar 30, 2024
f103f18
chore(main): release 6.63.0 (#2985)
release-please[bot] Mar 30, 2024
a8531fd
chore(main): release 6.63.1-SNAPSHOT (#2991)
release-please[bot] Mar 30, 2024
e74319c
chore: clean up some warnings and malformed comments (#2977)
olavloite Apr 1, 2024
248744b
chore(deps): update dependency com.google.cloud:google-cloud-spanner …
renovate-bot Apr 1, 2024
629fb32
feat: add endpoint connection URL property (#2969)
olavloite Apr 2, 2024
b72581c
feat: support max_commit_delay in Connection API (#2954)
olavloite Apr 2, 2024
d05695b
chore: minor improvements to default benchmarks. (#2993)
arpan14 Apr 2, 2024
417a83c
build(deps): update dependency org.jacoco:jacoco-maven-plugin to v0.8…
renovate-bot Apr 3, 2024
2d6f4f3
chore: add regex to match unmanaged dependency check (#1941) (#2971)
gcf-owl-bot[bot] Apr 3, 2024
d2b4862
feat: Add SessionPoolOptions, SpannerOptions protos in executor proto…
gcf-owl-bot[bot] Apr 3, 2024
b8074b5
Merge branch 'main' into pg-oid
tlhquynh Apr 3, 2024
cdd1c91
Merge branch 'main' into pg-oid
tlhquynh Apr 4, 2024
b63740d
chore: Remove unused CLIRR entries
tlhquynh Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .java-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
17.0.9
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this file (and potentially add the file name to .gitignore to prevent it from being included in commits in the future)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

4 changes: 4 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: Remove file from GitHub and add the filename to .gitignore

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"java.jdt.ls.vmargs": "-XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xmx2G -Xms100m -Xlog:disable",
"java.compile.nullAnalysis.mode": "automatic"
}
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,13 @@ implementation 'com.google.cloud:google-cloud-spanner'
If you are using Gradle without BOM, add this to your dependencies:

```Groovy
implementation 'com.google.cloud:google-cloud-spanner:6.61.0'
implementation 'com.google.cloud:google-cloud-spanner:6.62.0'
```

If you are using SBT, add this to your dependencies:

```Scala
libraryDependencies += "com.google.cloud" % "google-cloud-spanner" % "6.61.0"
libraryDependencies += "com.google.cloud" % "google-cloud-spanner" % "6.62.0"
```
<!-- {x-version-update-end} -->

Expand Down Expand Up @@ -650,7 +650,7 @@ Java is a registered trademark of Oracle and/or its affiliates.
[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-spanner/java11.html
[stability-image]: https://img.shields.io/badge/stability-stable-green
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-spanner.svg
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-spanner/6.61.0
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-spanner/6.62.0
[authentication]: https://github.com/googleapis/google-cloud-java#authentication
[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes
[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles
Expand Down
57 changes: 57 additions & 0 deletions google-cloud-spanner/clirr-ignored-differences.xml
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,63 @@
<className>com/google/cloud/spanner/StructReader</className>
<method>java.util.List getPgJsonbList(java.lang.String)</method>
</difference>
<!-- PG OID -->
<difference>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not be necessary to add the internal methods to this file, if those methods are given a default implementation that just throws UnsupportedOperationException

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These entries in clirr-ignored-differences.xml for StructReader can be removed, as we don't have those methods anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

final nit (I promise :-)): I think that the two remaining additions to this file for getPgOid() and getPgOidArray() can also be removed, as these methods are no longer added to the Value class.

<differenceType>7013</differenceType>
<className>com/google/cloud/spanner/AbstractStructReader</className>
<method>long[] getPgOidArrayInternal(int)</method>
</difference>
<difference>
<differenceType>7013</differenceType>
<className>com/google/cloud/spanner/AbstractStructReader</className>
<method>long getPgOidInternal(int)</method>
</difference>
<difference>
<differenceType>7013</differenceType>
<className>com/google/cloud/spanner/AbstractStructReader</className>
<method>java.util.List getPgOidListInternal(int)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>long getPgOid(int)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>long getPgOid(java.lang.String)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>long[] getPgOidArray(int)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>long[] getPgOidArray(java.lang.String)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>java.util.List getPgOidList(int)</method>
</difference>
<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
<method>java.util.List getPgOidList(java.lang.String)</method>
</difference>
<difference>
<differenceType>7013</differenceType>
<className>com/google/cloud/spanner/Value</className>
<method>long getPgOid()</method>
</difference>
<difference>
<differenceType>7013</differenceType>
<className>com/google/cloud/spanner/Value</className>
<method>java.util.List getPgOidArray()</method>
</difference>

<difference>
<differenceType>7012</differenceType>
<className>com/google/cloud/spanner/StructReader</className>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,31 @@ Double get(double[] array, int i) {
}
}

static class PgOidArray extends PrimitiveArray<Long, long[]> {
PgOidArray(ListValue protoList) {
super(protoList);
}

PgOidArray(long[] data, BitSet nulls) {
super(data, nulls, data.length);
}

@Override
long[] newArray(int size) {
return new long[size];
}

@Override
void setProto(long[] array, int i, com.google.protobuf.Value protoValue) {
array[i] = Long.parseLong(protoValue.getStringValue());
}

@Override
Long get(long[] array, int i) {
return array[i];
}
}

protected abstract GrpcStruct currRow();

@Override
Expand Down Expand Up @@ -400,6 +425,11 @@ protected String getPgJsonbInternal(int columnIndex) {
return currRow().getPgJsonbInternal(columnIndex);
}

@Override
protected long getPgOidInternal(int columnIndex) {
return currRow().getPgOidInternal(columnIndex);
}

@Override
protected ByteArray getBytesInternal(int columnIndex) {
return currRow().getBytesInternal(columnIndex);
Expand Down Expand Up @@ -480,6 +510,16 @@ protected List<String> getPgJsonbListInternal(int columnIndex) {
return currRow().getJsonListInternal(columnIndex);
}

@Override
protected long[] getPgOidArrayInternal(int columnIndex) {
return currRow().getPgOidArrayInternal(columnIndex);
}

@Override
protected List<Long> getPgOidListInternal(int columnIndex) {
return currRow().getPgOidListInternal(columnIndex);
}

@Override
protected List<ByteArray> getBytesListInternal(int columnIndex) {
return currRow().getBytesListInternal(columnIndex);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ protected String getPgJsonbInternal(int columnIndex) {
throw new UnsupportedOperationException("Not implemented");
}

protected abstract long getPgOidInternal(int columnIndex);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a default implementation for this and the other methods that are being added to this class. The default implementation should just throw UnsupportedOperationException. This prevents this change from being a breaking change, as anyone who has a class that extends AbstractStructReader does not get a compile error.

That should also remove the requirement to add the method names to clirr-ignored-differences.xml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


protected abstract ByteArray getBytesInternal(int columnIndex);

protected abstract Timestamp getTimestampInternal(int columnIndex);
Expand Down Expand Up @@ -122,6 +124,10 @@ protected List<String> getPgJsonbListInternal(int columnIndex) {
throw new UnsupportedOperationException("Not implemented");
}

protected abstract long[] getPgOidArrayInternal(int columnIndex);

protected abstract List<Long> getPgOidListInternal(int columnIndex);

protected abstract List<ByteArray> getBytesListInternal(int columnIndex);

protected abstract List<Timestamp> getTimestampListInternal(int columnIndex);
Expand Down Expand Up @@ -259,6 +265,19 @@ public String getPgJsonb(String columnName) {
return getPgJsonbInternal(columnIndex);
}

@Override
public long getPgOid(int columnIndex) {
checkNonNullOfType(columnIndex, Type.pgOid(), columnIndex);
return getPgOidInternal(columnIndex);
}

@Override
public long getPgOid(String columnName) {
int columnIndex = getColumnIndex(columnName);
checkNonNullOfType(columnIndex, Type.pgOid(), columnName);
return getPgOidInternal(columnIndex);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth considering not adding any of this at all, and instead instruct users that they have to get PG.OID values in the same way as INT64? So just call getLong(..) and the other corresponding methods. This is otherwise all just a copy-paste of those methods, and there's no additional validation of the values that we are getting.
I know that it is not what we have done for other types (e.g. we could in theory have done the same for JSON by telling users to just call getString(...)). But this might be the right time to start to prevent us from doing double work for future types, while it does not add any additional type safety or features for users.

@skuruppu Any thoughts on this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
I had an offline chat with @skuruppu and treating OID as Long is actually what we are doing for the other clients. I have removed all external PG.OID getters.


@Override
public ByteArray getBytes(int columnIndex) {
checkNonNullOfCodes(columnIndex, Arrays.asList(Code.PROTO, Code.BYTES), columnIndex);
Expand Down Expand Up @@ -505,6 +524,32 @@ public List<String> getPgJsonbList(String columnName) {
return getPgJsonbListInternal(columnIndex);
}

@Override
public long[] getPgOidArray(int columnIndex) {
checkNonNullOfType(columnIndex, Type.array(Type.pgOid()), columnIndex);
return getPgOidArrayInternal(columnIndex);
}

@Override
public long[] getPgOidArray(String columnName) {
int columnIndex = getColumnIndex(columnName);
checkNonNullOfType(columnIndex, Type.array(Type.pgOid()), columnName);
return getPgOidArrayInternal(columnIndex);
}

@Override
public List<Long> getPgOidList(int columnIndex) {
checkNonNullOfType(columnIndex, Type.array(Type.pgOid()), columnIndex);
return getPgOidListInternal(columnIndex);
}

@Override
public List<Long> getPgOidList(String columnName) {
int columnIndex = getColumnIndex(columnName);
checkNonNullOfType(columnIndex, Type.array(Type.pgOid()), columnName);
return getPgOidListInternal(columnIndex);
}

@Override
public List<ByteArray> getBytesList(int columnIndex) {
checkNonNullOfCodes(columnIndex, Collections.singletonList(Code.ARRAY), columnIndex);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,18 @@ public String getPgJsonb(String columnName) {
return delegate.get().getPgJsonb(columnName);
}

@Override
public long getPgOid(int columnIndex) {
checkValidState();
return delegate.get().getPgOid(columnIndex);
}

@Override
public long getPgOid(String columnName) {
checkValidState();
return delegate.get().getPgOid(columnName);
}

@Override
public ByteArray getBytes(int columnIndex) {
checkValidState();
Expand Down Expand Up @@ -373,6 +385,29 @@ public List<String> getPgJsonbList(String columnName) {
return delegate.get().getPgJsonbList(columnName);
}

public long[] getPgOidArray(int columnIndex) {
checkValidState();
return delegate.get().getPgOidArray(columnIndex);
}

@Override
public long[] getPgOidArray(String columnName) {
checkValidState();
return delegate.get().getPgOidArray(columnName);
}

@Override
public List<Long> getPgOidList(int columnIndex) {
checkValidState();
return delegate.get().getPgOidList(columnIndex);
}

@Override
public List<Long> getPgOidList(String columnName) {
checkValidState();
return delegate.get().getPgOidList(columnName);
}

@Override
public List<ByteArray> getBytesList(int columnIndex) {
checkValidState();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import com.google.cloud.spanner.AbstractResultSet.Float64Array;
import com.google.cloud.spanner.AbstractResultSet.Int64Array;
import com.google.cloud.spanner.AbstractResultSet.LazyByteArray;
import com.google.cloud.spanner.AbstractResultSet.PgOidArray;
import com.google.cloud.spanner.Type.Code;
import com.google.cloud.spanner.Type.StructField;
import com.google.common.base.Preconditions;
Expand Down Expand Up @@ -114,6 +115,9 @@ private Object writeReplace() {
case PG_JSONB:
builder.set(fieldName).to(Value.pgJsonb((String) value));
break;
case PG_OID:
builder.set(fieldName).to(Value.pgOid((Long) value));
break;
case BYTES:
builder
.set(fieldName)
Expand Down Expand Up @@ -158,6 +162,9 @@ private Object writeReplace() {
case PG_JSONB:
builder.set(fieldName).toPgJsonbArray((Iterable<String>) value);
break;
case PG_OID:
builder.set(fieldName).toPgOidArray((Iterable<Long>) value);
break;
case BYTES:
case PROTO:
builder
Expand Down Expand Up @@ -262,6 +269,7 @@ private static Object decodeValue(Type fieldType, com.google.protobuf.Value prot
checkType(fieldType, proto, KindCase.BOOL_VALUE);
return proto.getBoolValue();
case INT64:
case PG_OID:
case ENUM:
checkType(fieldType, proto, KindCase.STRING_VALUE);
return Long.parseLong(proto.getStringValue());
Expand Down Expand Up @@ -339,6 +347,8 @@ static Object decodeArrayValue(Type elementType, ListValue listValue) {
case STRUCT:
case PROTO:
return Lists.transform(listValue.getValuesList(), input -> decodeValue(elementType, input));
case PG_OID:
return new PgOidArray(listValue);
default:
throw new AssertionError("Unhandled type code: " + elementType.getCode());
}
Expand Down Expand Up @@ -460,6 +470,12 @@ protected String getPgJsonbInternal(int columnIndex) {
return (String) rowData.get(columnIndex);
}

@Override
protected long getPgOidInternal(int columnIndex) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove and rely on getLongInternal instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

ensureDecoded(columnIndex);
return (Long) rowData.get(columnIndex);
}

@Override
protected ByteArray getBytesInternal(int columnIndex) {
ensureDecoded(columnIndex);
Expand Down Expand Up @@ -563,6 +579,8 @@ protected Value getValueInternal(int columnIndex) {
return Value.json(isNull ? null : getJsonInternal(columnIndex));
case PG_JSONB:
return Value.pgJsonb(isNull ? null : getPgJsonbInternal(columnIndex));
case PG_OID:
return Value.pgOid(isNull ? null : getPgOidInternal(columnIndex));
case BYTES:
return Value.internalBytes(isNull ? null : getLazyBytesInternal(columnIndex));
case PROTO:
Expand Down Expand Up @@ -598,6 +616,8 @@ protected Value getValueInternal(int columnIndex) {
return Value.jsonArray(isNull ? null : getJsonListInternal(columnIndex));
case PG_JSONB:
return Value.pgJsonbArray(isNull ? null : getPgJsonbListInternal(columnIndex));
case PG_OID:
return Value.pgOidArray(isNull ? null : getPgOidListInternal(columnIndex));
case BYTES:
return Value.bytesArray(isNull ? null : getBytesListInternal(columnIndex));
case PROTO:
Expand Down Expand Up @@ -771,6 +791,18 @@ protected List<String> getPgJsonbListInternal(int columnIndex) {
return Collections.unmodifiableList((List<String>) rowData.get(columnIndex));
}

@Override
protected PgOidArray getPgOidListInternal(int columnIndex) {
ensureDecoded(columnIndex);
return (PgOidArray) rowData.get(columnIndex);
}

@Override
protected long[] getPgOidArrayInternal(int columnIndex) {
ensureDecoded(columnIndex);
return getPgOidListInternal(columnIndex).toPrimitiveArray(columnIndex);
}

@Override
@SuppressWarnings("unchecked") // We know ARRAY<BYTES> produces a List<LazyByteArray>.
protected List<ByteArray> getBytesListInternal(int columnIndex) {
Expand Down
Loading
Loading