-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: enable session leaks prevention by cleaning up long-running tra… #2655
Changes from all commits
93fe1e0
6db9260
b1298ee
ee6e0c3
c39585f
0a65b2c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -281,3 +281,84 @@ This will cause the following to happen internally in the client library: | |
1. The `TransactionRunner` will automatically commit the transaction if the supplied user code | ||
finished without any errors. The `Commit` RPC that is invoked uses a thread from the default gRPC | ||
thread pool. | ||
|
||
### Session Leak | ||
A `DatabaseClient` object of the Client Library has a limit on the number of maximum sessions. For example the | ||
default value of `MaxSessions` in the Java Client Library is 400. You can configure these values at the time of | ||
creating a `Spanner` instance by setting custom `SessionPoolOptions`. When all the sessions are checked | ||
out of the session pool, every new transaction has to wait until a session is returned to the pool. | ||
If a session is never returned to the pool (hence causing a session leak), the transactions will have to wait | ||
indefinitely and your application will be blocked. | ||
|
||
#### Common Root Causes | ||
The most common reason for session leaks in the Java client library are: | ||
1. Not closing a `ResultSet` that is returned by `executeQuery`. Always put `ResultSet` objects in a try-with-resources block, or take other measures to ensure that the `ResultSet` is always closed. | ||
2. Not closing a `ReadOnlyTransaction` when you no longer need it. Always put `ReadOnlyTransaction` objects in a try-with-resources block, or take other measures to ensure that the `ReadOnlyTransaction` is always closed. | ||
3. Not closing a `TransactionManager` when you no longer need it. Always put `TransactionManager` objects in a try-with-resources block, or take other measures to ensure that the `TransactionManager` is always closed. | ||
|
||
As shown in the example below, the `try-with-resources` block releases the session after it is complete. | ||
If you don't use `try-with-resources` block, unless you explicitly call the `close()` method on all resources | ||
such as `ResultSet`, the session is not released back to the pool. | ||
|
||
```java | ||
DatabaseClient client = | ||
spanner.getDatabaseClient(DatabaseId.of("my-project", "my-instance", "my-database")); | ||
try (ResultSet resultSet = | ||
client.singleUse().executeQuery(Statement.of("select col1, col2 from my_table"))) { | ||
while (resultSet.next()) { | ||
// use the results. | ||
} | ||
} | ||
``` | ||
|
||
#### Debugging and Resolving Session Leaks | ||
|
||
##### Logging | ||
Enabled by default, the logging option shares warn logs when you have exhausted >95% of your session pool. | ||
This could mean two things, either you need to increase the max sessions in your session pool (as the number | ||
of queries run using the client side database object is greater than your session pool can serve) or you may | ||
have a session leak. | ||
|
||
To help debug which transactions may be causing this session leak, the logs will also contain stack traces of | ||
transactions which have been running longer than expected. The logs are pushed to a destination based on | ||
how the log exporter is configured for the host application. | ||
|
||
Comment on lines
+322
to
+325
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we logging in the Java client? Based on the comment above, I would expect 'No'. If we are logging: I'm pretty sure that we are not pushing those logs to any 'destination based on how the log exporter is configured'. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some of the other client libraries such as Node/Python, emit an additional log every-time the application uses > 95% of the sessions. In Java, we use the condition as a pre-condition for the algo but don't emit any log for this condition. We only emit one of the below logs, Now what I mean as pushing log to the exporter is - the logs will be stored in a destination as per what is configured for the application. For ex - Some apps may configure cloud logging while others could configure logs to be stored on separate files on the app server. The note about the exporter is not too relevant for Java where there is a default logging behaviour and I think it was more relevant for other languages like Node where a default logging capability was absent. We added a custom logging capability in Node (as part of sessions leaks) where the default logging was console logs or you could define some other exporter. LMK if this documentation can be better framed or excluded in context of Java client. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We're using |
||
``` java | ||
final SessionPoolOptions sessionPoolOptions = | ||
SessionPoolOptions.newBuilder().setWarnIfInactiveTransactions().build() | ||
|
||
final Spanner spanner = | ||
SpannerOptions.newBuilder() | ||
.setSessionPoolOption(sessionPoolOptions) | ||
.build() | ||
.getService(); | ||
final DatabaseClient client = spanner.getDatabaseClient(databaseId); | ||
|
||
// Example Log message to warn presence of long running transactions | ||
// Detected long-running session <session-info>. To automatically remove long-running sessions, set SessionOption ActionOnInactiveTransaction | ||
// to WARN_AND_CLOSE by invoking setWarnAndCloseIfInactiveTransactions() method. <Stack Trace and information on session> | ||
|
||
``` | ||
##### Automatically clean inactive transactions | ||
When the option to automatically clean inactive transactions is enabled, the client library will automatically spot | ||
problematic transactions that are running for extremely long periods of time (thus causing session leaks) and close them. | ||
The session will be removed from the pool and be replaced by a new session. To dig deeper into which transactions are being | ||
closed, you can check the logs to see the stack trace of the transactions which might be causing these leaks and further | ||
debug them. | ||
|
||
``` java | ||
final SessionPoolOptions sessionPoolOptions = | ||
SessionPoolOptions.newBuilder().setWarnAndCloseIfInactiveTransactions().build() | ||
|
||
final Spanner spanner = | ||
SpannerOptions.newBuilder() | ||
.setSessionPoolOption(sessionPoolOptions) | ||
.build() | ||
.getService(); | ||
final DatabaseClient client = spanner.getDatabaseClient(databaseId); | ||
|
||
// Example Log message for when transaction is recycled | ||
// Removing long-running session <Stack Trace and information on session> | ||
``` | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think for java we won't have this log right so we can remove the line?