py_test with unittest.TestCase doesn't work with sharding

[meteorcloudy]

Description of the bug:

The shard_count attribute of py_test doesn't actually do any sharding if you just use unittest.TestCase, instead it just causes the test to run multiple tests.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

In the Bazel repo:

• Add shard_count = 5 to //examples/py_native:test
• Run bazel test //examples/py_native:test
• Check the test logs under bazel-testlogs/examples/py_native/test/

Every test log has:

Ran 2 tests in 0.000s

OK

Which operating system are you running Bazel on?

all

What is the output of bazel info release?

any

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

• absltest supports sharding, switching to this framework may make it work.
• https://bazel.build/reference/test-encyclopedia#test-sharding is not true, Bazel doesn't check TEST_SHARD_STATUS_FILE to avoid launching additional test runners when sharding is not supported.

[meteorcloudy]

I can confirm switching to absltest enables sharding correctly:

pcloudy@pcloudy-macbookpro3:~/workspace/bazel (master)
$ cat bazel-testlogs/examples/py_native/test/shard_1_of_5/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/py_native:test
-----------------------------------------------------------------------------
Running tests under Python 3.9.6: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/4/execroot/io_bazel/bazel-out/darwin_arm64-fastbuild/bin/examples/py_native/test.runfiles/io_bazel/examples/py_native/test.py:15: DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(Fib(5), 8)
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
pcloudy@pcloudy-macbookpro3:~/workspace/bazel (master)
$ cat bazel-testlogs/examples/py_native/test/shard_2_of_5/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/py_native:test
-----------------------------------------------------------------------------
Running tests under Python 3.9.6: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/3/execroot/io_bazel/bazel-out/darwin_arm64-fastbuild/bin/examples/py_native/test.runfiles/io_bazel/examples/py_native/test.py:12: DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(GetNumber(), 42)
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
pcloudy@pcloudy-macbookpro3:~/workspace/bazel (master)
$ cat bazel-testlogs/examples/py_native/test/shard_3_of_5/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/py_native:test
-----------------------------------------------------------------------------
Running tests under Python 3.9.6: /Applications/Xcode.app/Contents/Developer/usr/bin/python3

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
pcloudy@pcloudy-macbookpro3:~/workspace/bazel (master)
$ cat bazel-testlogs/examples/py_native/test/shard_4_of_5/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/py_native:test
-----------------------------------------------------------------------------
Running tests under Python 3.9.6: /Applications/Xcode.app/Contents/Developer/usr/bin/python3

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
pcloudy@pcloudy-macbookpro3:~/workspace/bazel (master)
$ cat bazel-testlogs/examples/py_native/test/shard_5_of_5/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/py_native:test
-----------------------------------------------------------------------------
Running tests under Python 3.9.6: /Applications/Xcode.app/Contents/Developer/usr/bin/python3

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

[fmeum]

@meteorcloudy I noticed the lack of an mtime check before while working in rules_jvm's JUnit 5 runner. This seems quite error-prone as we saw here.

Do you think I could reasonably add that check for Bazel 7? As migrating only requires dropping shard_count and enjoying faster tests, that sounds like an acceptable "breaking" change to me. Edit: Noticed that it could be more breaking if there are non-compliant runners out there, will investigate.

[Wyverald]

If by "mtime check" you mean the TEST_SHARD_STATUS_FILE thing, then yeah I think adding it for Bazel 7 is very reasonable. If the runner doesn't touch that file, we can simply fail the test, and the fix is very easy -- just remove shard_count.

[meteorcloudy]

I'll recommend to add the check, but guard it behind an incompatible flag, then we can even cherry pick it back to 6.x (off by default) so that users can start migrate earlier.

[meteorcloudy]

The Bazel bug has been fixed by #18236 and #18469