Change detection random errors despite CI passing (most of the times) #623

shahzadlone · 2022-07-12T08:02:14Z

Change detection randomly errors out despite CI passing most of the time.

Specifically talking about this rule: make test:changes.

Currently in .circleci/config.yml file we have: gotestsum --junitfile /tmp/test-reports/unit-tests.xml -- ./... -p 1

and in the Makefile we have: env DEFRA_DETECT_DATABASE_CHANGES=true go test ./... -p 1

as part of #619 I changed the Makefile rule to:
env DEFRA_DETECT_DATABASE_CHANGES=true gotestsum --junitfile /tmp/defradb-dev/changes.xml -- ./... -p 1

which would be called by the CI.

This is where the errors become more prominent (even though they were happening before too).

Here is one of the errors I encountered (TempDir RemoveAll cleanup: open /tmp: too many open files):

$ make test:changes

env DEFRA_DETECT_DATABASE_CHANGES=true gotestsum --junitfile /tmp/defradb-dev/changes.xml -- ./... -p 1
   api/http (44ms)
   cli
   client (4ms)
   cmd/defradb
   cmd/genclidocs
   cmd/genmanpages
   config (24ms)
   connor
   connor/numbers
   core (4ms)
   core/crdt (5ms)
   core/net
   datastore
   datastore/badger/v3
   datastore/iterable
   db (82ms)
   db/base
   db/container
   db/fetcher
   logging (1.609s)
   merkle/clock (6ms)
   merkle/crdt (6ms)
   net
   net/api
   net/api/client
   net/api/pb
   net/pb
   net/utils
   node (36ms)
   query/graphql/mapper
   query/graphql/parser (3ms)
   query/graphql/parser/types
   query/graphql/planner
   query/graphql/schema (15ms)
   query/graphql/schema/types
   tests/bench
   tests/bench/collection (4.479s)
   tests/bench/fixtures
   tests/bench/query/planner (866ms)
   tests/bench/query/simple (888ms)
   tests/bench/storage (872ms)
   tests/integration
   tests/integration/collection/update (2.247s)
   tests/integration/mutation/inline_array
   tests/integration/mutation/inline_array/update (3.421s)
   tests/integration/mutation/relation
   tests/integration/mutation/relation/delete (2.203s)
   tests/integration/mutation/simple
   tests/integration/mutation/simple/create (4.01s)
   tests/integration/mutation/simple/delete (10.384s)
   tests/integration/mutation/simple/mix (936ms)
   tests/integration/mutation/simple/update (5.31s)
   tests/integration/query/all_commits (9.003s)
   tests/integration/query/commit (4.792s)
   tests/integration/query/complex (2.235s)
   tests/integration/query/explain (21.319s)
   tests/integration/query/inline_array (14.739s)
   tests/integration/query/latest_commits (5.545s)
   tests/integration/query/one_to_many (14.111s)
   tests/integration/query/one_to_many_multiple (7.215s)
   tests/integration/query/one_to_one (4.275s)
   tests/integration/query/one_to_two_many (2.838s)
   tests/integration/query/simple (1m21.618s)
   tests/integration/query/simple/with_filter (16.366s)

=== Skipped
=== SKIP: tests/integration/mutation/simple/mix TestMutationWithTxnDeletesUserGivenSameTransaction (0.00s)

=== SKIP: tests/integration/mutation/simple/mix TestMutationWithTxnDoesNotDeletesUserGivenDifferentTransactions (0.00s)

=== SKIP: tests/integration/mutation/simple/mix TestMutationWithTxnDoesUpdateUserGivenSameTransactions (0.00s)

=== SKIP: tests/integration/mutation/simple/mix TestMutationWithTxnDoesNotUpdateUserGivenDifferentTransactions (0.00s)

=== SKIP: tests/integration/mutation/simple/mix TestMutationWithTxnDoesNotAllowUpdateInSecondTransactionUser (0.00s)

=== SKIP: tests/integration/query/one_to_many TestQueryOneToManyWithUnknownCidAndDocKey (0.00s)
    with_cid_dockey_test.go:66:

=== Failed
=== FAIL: tests/integration/query/simple TestQuerySimpleWithNumericOrderDescendingAndBooleanOrderAscending (0.85s)
2022-07-12T02:18:33.929-0400, WARN, defra.tests.integration, Simple query with compound order, {"Database": "badger-file-system"}
2022-07-12T02:18:33.929-0400, INFO, defra.db, Closing DefraDB process...
2022-07-12T02:18:33.945-0400, INFO, defra.db, Successfully closed running process
2022-07-12T02:18:34.763-0400, WARN, defra.tests.integration, , {"QueryResults": [{"Age":55,"Name":"Carlo","Verified":true},{"Age":21,"Name":"Bob","Verified":false},{"Age":21,"Name":"John","Verified":true},{"Age":19,"Name":"Alice","Verified":false}]}
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithSumWithFilter (0.00s)
    utils.go:653:
                Error Trace:    utils.go:653
                                                        utils.go:277
                                                        utils.go:32
                                                        with_sum_filter_test.go:48
                Error:          Received unexpected error:
                                cannot open directory "/tmp/TestQuerySimpleWithSumWithFilter4250419997/001" error: open /tmp/TestQuerySimpleWithSumWithFilter4250419997/001: too many open files
                Test:           TestQuerySimpleWithSumWithFilter
                Messages:       Simple query, sum with filter
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_sum_filter_test.go:48
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithSumWithFilter
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithSumOnUndefined (0.00s)
    utils.go:657:
                Error Trace:    utils.go:657
                                                        utils.go:277
                                                        utils.go:32
                                                        with_sum_test.go:28
                Error:          Target error should be in err chain:
                                expected: "Aggregate must be provided with a property to aggregate."
                                in chain: "cannot open directory \"/tmp/TestQuerySimpleWithSumOnUndefined395478893/001\" error: open /tmp/TestQuerySimpleWithSumOnUndefined395478893/001: too many open files"
                Test:           TestQuerySimpleWithSumOnUndefined
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_sum_test.go:28
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithSumOnUndefined
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithSumOnEmptyCollection (0.00s)
    utils.go:653:
                Error Trace:    utils.go:653
                                                        utils.go:277
                                                        utils.go:32
                                                        with_sum_test.go:44
                Error:          Received unexpected error:
                                cannot open directory "/tmp/TestQuerySimpleWithSumOnEmptyCollection1480651534/001" error: open /tmp/TestQuerySimpleWithSumOnEmptyCollection1480651534/001: too many open files
                Test:           TestQuerySimpleWithSumOnEmptyCollection
                Messages:       Simple query, sum on empty
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_sum_test.go:44
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithSumOnEmptyCollection
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithSum (0.00s)
    utils.go:653:
                Error Trace:    utils.go:653
                                                        utils.go:277
                                                        utils.go:32
                                                        with_sum_test.go:72
                Error:          Received unexpected error:
                                cannot open directory "/tmp/TestQuerySimpleWithSum2428078486/001" error: open /tmp/TestQuerySimpleWithSum2428078486/001: too many open files
                Test:           TestQuerySimpleWithSum
                Messages:       Simple query, sum
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_sum_test.go:72
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithSum
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithEmbeddedLatestCommit (0.00s)
    utils.go:653:
                Error Trace:    utils.go:653
                                                        utils.go:277
                                                        utils.go:32
                                                        with_version_test.go:65
                Error:          Received unexpected error:
                                cannot open directory "/tmp/TestQuerySimpleWithEmbeddedLatestCommit1952084130/001" error: open /tmp/TestQuerySimpleWithEmbeddedLatestCommit1952084130/001: too many open files
                Test:           TestQuerySimpleWithEmbeddedLatestCommit
                Messages:       Embedded latest commits query within object query
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_version_test.go:65
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithEmbeddedLatestCommit
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

=== FAIL: tests/integration/query/simple TestQuerySimpleWithMultipleAliasedEmbeddedLatestCommit (0.00s)
    utils.go:653:
                Error Trace:    utils.go:653
                                                        utils.go:277
                                                        utils.go:32
                                                        with_version_test.go:125
                Error:          Received unexpected error:
                                cannot open directory "/tmp/TestQuerySimpleWithMultipleAliasedEmbeddedLatestCommit882131000/001" error: open /tmp/TestQuerySimpleWithMultipleAliasedEmbeddedLatestCommit882131000/001: too many open files
                Test:           TestQuerySimpleWithMultipleAliasedEmbeddedLatestCommit
                Messages:       Embedded, aliased, latest commits query within object query
    utils.go:280:
                Error Trace:    utils.go:280
                                                        utils.go:32
                                                        with_version_test.go:125
                Error:          Should NOT be empty, but was []
                Test:           TestQuerySimpleWithMultipleAliasedEmbeddedLatestCommit
    testing.go:967: TempDir RemoveAll cleanup: open /tmp: too many open files

DONE 572 tests, 6 skipped, 7 failures in 234.266s

The text was updated successfully, but these errors were encountered:

orpheuslummis · 2022-07-12T11:37:52Z

What do you obtain running ulimit -n on your system?

orpheuslummis · 2022-07-12T11:40:04Z

For fun, as of Go 1.19 that's the file part of os package that "raises the limit unconditionally" to its maximum. https://cs.opensource.google/go/go/+/master:src/os/rlimit.go

shahzadlone · 2022-07-12T15:09:29Z

What do you obtain running ulimit -n on your system?

1024

orpheuslummis · 2022-07-12T20:14:29Z

We will switch to 1.19 in early 2023 and that might solve the issue, if the go stdlib implementation works on WSL.

There is the possibility to backport the file limit raising that Go 1.19 implemented directly in defradb.

You can also quick fix your system using ulimit -n unlimited, replacing unlimited by a higher number if needed.

I would expect that defradb itself doesn't use a very high number of files, contrary to its tests.

shahzadlone added bug Something isn't working area/testing Related to any test or testing suite ci/build This is issue is about the build or CI system, and the administration of it. labels Jul 12, 2022

shahzadlone mentioned this issue Jul 12, 2022

tools: Use gotestsum over go test #619

Merged

4 tasks

orpheuslummis mentioned this issue Jul 12, 2022

fix: Increase system's open file limit in integration tests #627

Merged

4 tasks

orpheuslummis closed this as completed in #627 Jul 13, 2022

shahzadlone mentioned this issue Jul 15, 2022

Change detection random errors continue (caught with -race) #637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change detection random errors despite CI passing (most of the times) #623

Change detection random errors despite CI passing (most of the times) #623

shahzadlone commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022

shahzadlone commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022

Change detection random errors despite CI passing (most of the times) #623

Change detection random errors despite CI passing (most of the times) #623

Comments

shahzadlone commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022

shahzadlone commented Jul 12, 2022

orpheuslummis commented Jul 12, 2022