Sqlite tests with transactions are flaky

[smitpatel]

I am hitting following errors intermittently when running build.cmd on command line. Re-running gets rid of the error.

Assemblies Run 
Microsoft.EntityFrameworkCore.Sqlite.FunctionalTests.dll
Summary 
Tests run: 11781 — Errors: 1, Failures: 3, Skipped: 860, Run time: 101.251s, Finished: 06/24/2019 17:48:09

Errors 
✘
Test Class Cleanup (Microsoft.EntityFrameworkCore.InterceptionSqliteTest) 
Microsoft.Data.Sqlite.SqliteException : SQLite Error 1: 'cannot rollback - no transaction is active'.

Failed tests 
0.0380121s 
✘
 Microsoft.EntityFrameworkCore.InterceptionSqliteTest.Intercept_non_query_to_replace_execution(async: True, inject: True)
System.ArgumentOutOfRangeException : Index was out of range. Must be non-negative and less than the size of the collection.\r\nParameter name: index
   at System.Collections.Generic.List`1.RemoveAt(Int32 index)
   at Microsoft.Data.Sqlite.SqliteConnection.RemoveCommand(SqliteCommand command) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteConnection.cs:line 403
   at Microsoft.Data.Sqlite.SqliteCommand.Dispose(Boolean disposing) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteCommand.cs:line 209
   at System.ComponentModel.Component.Dispose()
   at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText, SqliteParameter[] parameters) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\Extensions\SqliteConnectionExtensions.cs:line 19
   at Microsoft.Data.Sqlite.SqliteTransaction.RollbackInternal() in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteTransaction.cs:line 146
   at Microsoft.Data.Sqlite.SqliteTransaction.Dispose(Boolean disposing) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteTransaction.cs:line 130
   at System.Data.Common.DbTransaction.Dispose()
   at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalTransaction.cs:line 159
   at Microsoft.EntityFrameworkCore.InterceptionTestBase`2.Intercept_non_query_to_replace_execution(Boolean async, Boolean inject) in D:\code\EntityFrameworkCore\test\EFCore.Relational.Specification.Tests\InterceptionTestBase.cs:line 648
--- End of stack trace from previous location where exception was thrown ---
0.0302253s 
✘
 Microsoft.EntityFrameworkCore.InterceptionSqliteTest.Intercept_non_query_with_two_injected_interceptors(async: False)
System.InvalidOperationException : SqliteConnection does not support nested transactions.
   at Microsoft.Data.Sqlite.SqliteConnection.BeginTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteConnection.cs:line 473
   at Microsoft.Data.Sqlite.SqliteConnection.BeginDbTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteConnection.cs:line 454
   at System.Data.Common.DbConnection.BeginTransaction(IsolationLevel isolationLevel)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionWithNoPreconditions(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 255
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 210
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction() in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 183
   at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.BeginTransaction() in D:\code\EntityFrameworkCore\src\EFCore\Infrastructure\DatabaseFacade.cs:line 145
   at Microsoft.EntityFrameworkCore.InterceptionTestBase`2.TestCompositeNonQueryInterceptors(UniverseContext context, Boolean async) in D:\code\EntityFrameworkCore\test\EFCore.Relational.Specification.Tests\InterceptionTestBase.cs:line 1211
   at Microsoft.EntityFrameworkCore.InterceptionTestBase`2.Intercept_non_query_with_two_injected_interceptors(Boolean async) in D:\code\EntityFrameworkCore\test\EFCore.Relational.Specification.Tests\InterceptionTestBase.cs:line 1267
--- End of stack trace from previous location where exception was thrown ---
0.0047849s 
✘
 Microsoft.EntityFrameworkCore.InterceptionSqliteTest.Intercept_non_query_with_two_injected_interceptors(async: True)
System.InvalidOperationException : SqliteConnection does not support nested transactions.
   at Microsoft.Data.Sqlite.SqliteConnection.BeginTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteConnection.cs:line 473
   at Microsoft.Data.Sqlite.SqliteConnection.BeginDbTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\Microsoft.Data.Sqlite.Core\SqliteConnection.cs:line 454
   at System.Data.Common.DbConnection.BeginTransaction(IsolationLevel isolationLevel)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionWithNoPreconditions(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 255
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction(IsolationLevel isolationLevel) in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 210
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction() in D:\code\EntityFrameworkCore\src\EFCore.Relational\Storage\RelationalConnection.cs:line 183
   at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.BeginTransaction() in D:\code\EntityFrameworkCore\src\EFCore\Infrastructure\DatabaseFacade.cs:line 145
   at Microsoft.EntityFrameworkCore.InterceptionTestBase`2.TestCompositeNonQueryInterceptors(UniverseContext context, Boolean async) in D:\code\EntityFrameworkCore\test\EFCore.Relational.Specification.Tests\InterceptionTestBase.cs:line 1211
   at Microsoft.EntityFrameworkCore.InterceptionTestBase`2.Intercept_non_query_with_two_injected_interceptors(Boolean async) in D:\code\EntityFrameworkCore\test\EFCore.Relational.Specification.Tests\InterceptionTestBase.cs:line 1267
--- End of stack trace from previous location where exception was thrown ---

[ajcvickers]

I believe the first failure shown is the real issue--seems to be a race condition in disposing a transaction that was not committed. The second two issues happen because the transaction on the shared connection failed in the first test.
/cc @bricelam

[ericsink]

Which version of SQLitePCLRaw is involved here? The 2.0 prerelease? Or 1.x?

[smitpatel]

@ericsink - 1.1.12

[bricelam]

@ajcvickers @smitpatel Have you seen this recently? (How much time should I spend investigating?)

[ajcvickers]

@bricelam Yes--I see it a few times per week when running locally. From a test flakiness perspective it doesn't need doing for 3.0, but it looks like a product bug. It's worth spending a couple of hours on I think.

[bricelam]

I'm only able to repro this when disposing the connection and transaction on different threads. Test issue?

[ajcvickers]

This is the test issue I most often see associated with this failure: #17017
/cc @AndriySvyryd

[smitpatel]

I haven't seen any association between both of them. Especially, they are also for different providers.

[ajcvickers]

@smitpatel Sometimes it happens with the SQL Server tests; sometimes with the SQLite tests. The message is different, but the symptom is the same--attempting to use an ambient transaction across threads. I suspect it has something to do with the way tests share a connection, but I haven't validated that.

[smitpatel]

Sorry, I interpreted as the other test issue had something to cause this rather than both have similar root cause.

[TharisSolutionsLtd]

Hello,

I am having this same issue, but I am only getting it from an IHostedService. The hosted service is using the DbContext after creating a scope from the service provider (all within "using" statements). As the hosted service is basically spamming the database, this is what seems to cause the issue. Without fail, it always errors when the method SqliteConnection.RemoveCommand is called in the stack trace, no matter the origin of the call (happens after SqliteCommand.Dispose). The error could happen from: DbContext.Database.EnsureCreated, DbContext.Set<>.Find or generally any query from the DbSet. The exception message is always "Object reference not set to an instance of the object", but there are no objects in my control that are null, it seems like an SQLite framework issue. I am using Microsoft.EntityFrameworkCore.Sqlite v3.1.0 in .NET Core 3.1.

Running other integration tests via the WebApplicationFactory and replacing the provider with SQLite runs fine and works as I expect - I am using SQLite for testing to take advantage of the relational constraints it provides unlike the EF core in memory solution.