Flank does not handle @SdkSuppress annotation correctly

[pawelpasterz]

Describe the bug

A clear and concise description of what the bug is.

copy-pasted from the original issue #818 (comment)

We're seeing this issue as well, usually because we have @SdkSuppress on on a test which happens to be the only test on the shard.

Although in our case, FTL doesn't detect that there were no tests in the shard - instead, it reports a single test with the name null as seen here:

java.lang.ClassNotFoundException: Invalid name: no tests found
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:400)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:72)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:105)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:804)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:575)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:393)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

We also don't see the message that you mentioned in the output, "One or more shards failed because there are no test cases inside." The table in the output just says that "N tests failed".

@bootstraponline Would this be considered the same issue, or should I file a different one? Or is this something that needs to be looked into on the FTL side? (Or even ATO?) Is this something that can be solved in Flank?

Expected behavior

A clear and concise description of what you expected to happen.

Flank should skip tests when sdk version requirement is not fulfilled
Once fixed -- update IT with new case

[bootstraponline]

@damianw hey, we discussed this issue in sprint planning today. The team will be working on this.

[damianw]

Thanks for the update! Looking forward to it.

[piotradamczyk5]

@damianw could you test with Flank v21.07.2 and let us know about the results?

[damianw]

Will do, thanks!