#7104 wdio-junit-reporter - Mark test as skipped if on retry

[lthurr]

Marking test as skipped in junit-reporter, and adding stacktrace if there is a failure

Proposed changes

This is meant to fix #7104
I did a little analysis in here, all the reporters behave the same, meaning if the test is retried is because it failed, so it will be marked as failed.
For junit, if a test is marked as failed (even tho it passed on the retry) then the whole suite is marked as failed.

So this PR is meant to fix that, it will mark the test as skipped and it will append the stack trace on the report.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• I have added proper type definitions for new commands (if appropriate)

Further comments

Let's suppose we retry two times and passes on the second time.

Junit XML: Before changes

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="suite" timestamp="..." time="0.008" tests="2" failures="1" errors="1" skipped="0">
    <properties>
      ...
    </properties>
    <testcase classname="..." name="test" time="0.007">
      <failure/>
      <error message="expect(received).toBe(expected) // Object.is equality;Expected: false;Received: true"/>
      <system-err><![CDATA[
Error: expect(received).toBe(expected) // Object.is equality[22m

Expected: false
Received: true
]]></system-err>
    </testcase>
    <testcase classname="" name="test" time="0">
    </testcase>
  </testsuite>
</testsuites>

Junit XML: After

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="suite" tests="2" failures="0" errors="0" skipped="1">
    <properties>
      ...
    </properties>
    <testcase classname="..." name="test" time="0.006">
      <skipped/>
      <system-err><![CDATA[
Error: expect(received).(expected) // Object.is equality[22m

Expected: false
Received: true
]]></system-err>
    </testcase>
    <testcase classname="..." name="test" time="0.001">
    </testcase>
  </testsuite>
</testsuites>

Reviewers: @webdriverio/project-committers

[christian-bromann]

Thank you for raising the PR.

Can we add a unit test for this?

[mgrybyk]

LGTM
Let's wait for unit tests to be added

[GPSC1]

Is this implemented for just "junit reporter" or all of the reporters??.. because i am facing this issue with "wdio-html-nice-reporter".
Issue- If there is 1 flaky test case which gets retried thrice and gets passed in 3rd attempt then repost shows failed tests count as 2 even when the test got passed on 3rd attempt.

wdio-html-nice-reporter version- 7.9.0

[christian-bromann]

Is this implemented for just "junit reporter" or all of the reporters??

Only for this reporter. Retries have to be handled by all reporter differently.

[GPSC1]

Ok.. Thanks for the info.. Is thr a way to resolve this issue if faced while using "wdio-html-nice-reporter"?