feat: show running builds in the Debug page #25936

warrensplayer · 2023-02-24T02:38:42Z

Closes Debug 1.1 - Show Running Build details #25759

Additional details

Before this update, a developer would have to wait for a run to complete before being able to see the failed tests that may be a part of the run. This change will allow the developer to switch to a running run and see any failed tests as they get recorded to the Cloud run.

Updates

Updated DebugNewRelevantRunBar
- Was moved above the Debug header when it is shown
- Link for "View run" was changed to "Switch to run"
- Link for "Switch to run" was added for a RUNNING run

Before

After

Added DebugTestingProgress component
- Shows when viewing a RUNNING run just below the header
- Indicates how many specs have completed with text as well as a progress bar
- Has a countdown when the run has a completionScheduledAt timestamp
- Switches to say completion is soon if countdown expires

NOTE I have not implemented the "Run Completion Delay" tooltip and indication at the end of the progress bar. I need to review with Product to see if we want to do this in a fast-follow

Updated DebugPendingRunSplash
- Removes the spec count because they are included in the DebugTestingProgress component
- Update copy to support run is currently processing specs and has a state when the run is scheduled to complete

Update DebugPageHeader
- Allow for the duration field to count up for a running run
Updated Sidebar badge to show when status of run is RUNNING
Updates to support animating the changing parts of the UI
- Updated the logic Debug.vue to cache and replace the data being passed to the DebugContainer to make sure the page does not flash when the query is rerunning to update the page
- Added various Transition and TransitionGroup components to fade components in and out
- Modified CSS and the structure of some components
  - Pulled internal padding from some components up to a parent container
  - Updated StatsMetadata to support a "prefix" slot so that the render in DebugSpec did not render a list (ul) within another list using incorrect DOM
Backend updates
- Updated moveToNext mutation to move to a specific run
- Updates to RelevantRunSpecsDataSource
  - Track overall test counts to know when to trigger a running run to refresh
  - Track new cloud field called scheduledToCompleteAt to know when to switch the page to the "scheduled to complete" state

Extra credit

Discovered a missing browser icon in LayeredBrowserIcons and added a placeholder. Filed a ticket to get the real icon added.
Update useDurationFormat to show milliseconds if the duration is less than 1 second

Steps to test

Component tests to review
- packages/app/src/debug/DebugContainer.cy.tsx
  - Added "renders running run with failed tests" test to some what mimic the phases of a running run
    - NOTE: Does not mock out DebugTestingProgress component due to issues with different mocked Urql queries interfering with each other
- packages/app/src/debug/DebugNewRelevantRunBar.cy.tsx
- packages/app/src/debug/DebugPageHeader.cy.tsx
- packages/app/src/debug/DebugPendingRunSplash.cy.tsx
- packages/app/src/debug/DebugTestingProgress.cy.tsx
- packages/app/src/debug/StatsMetadata.cy.tsx
- packages/app/src/navigation/SidebarNavigation.cy.tsx
E2E tests
- The debug.cy.ts tests started to fail after including the scheduledToCompleteAt field in the query from DebugContainer. When running locally, the page would thrash between states. I believe the cause was the mocked up responses in packages/app/cypress/fixtures/debug-Passing/gql-Debug.json and packages/app/cypress/fixtures/debug-Failing/gql-Debug.json not containing this field. Thus, Urql which see that it was not present and requery causing other queries to also rerun. After adding the scheduledToCompleteAt field, the tests started passing again.
- For this ticket, I relied on the CT tests for testing out the various states of the components and then live testing against a Cloud instance (staging and production). I worked on adding a new E2E tests for the running state, but could not get it working in short order so left it out.
Options for real world testing
- Setup a project on a local Cloud instance, on staging, or in production and manually record a run to it
- Check out the branch for this PR and trigger a Circle CI run against it to watch it.

How has the user experience changed?

See screenshots above

PR Tasks

Have tests been added/updated?
[-] Has a PR for user-facing changes been opened in cypress-documentation?
[-] Have API changes been updated in the type definitions?

cypress · 2023-02-24T03:07:04Z

25 flaky tests on run #44553 ↗︎

0	26874	1281	0	25

Details:

Merge branch 'develop' into stokes/25759_debug_running_builds
Project: cypress	Commit: `33f7cbe580`
Status: Passed	Duration: 18:30 💡
Started: Mar 6, 2023 7:15 PM	Ended: Mar 6, 2023 7:34 PM

commands/net_stubbing.cy.ts • 1 flaky test • 5x-driver-firefox

View Output Video

Test		Artifacts
network stubbing > intercepting request > can delay and throttle a StaticResponse		`Output`

cypress/cypress.cy.js • 3 flaky tests • 5x-driver-firefox

View Output Video

Test		Artifacts
... > correctly returns currentRetry		`Output`
... > correctly returns currentRetry		`Output`
... > correctly returns currentRetry		`Output`

specs_list_latest_runs.cy.ts • 2 flaky tests • app-e2e

View Output Video

Test		Artifacts
App/Cloud Integration - Latest runs and Average duration > when no runs are recorded > shows placeholders for all visible specs		`Output` `Screenshots` `Video`
App/Cloud Integration - Latest runs and Average duration > when offline > shows offline alert then hides it after coming online		`Output` `Screenshots` `Video`

cypress-origin-communicator.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
Cypress In Cypress Origin Communicator > cy.origin passivity with app interactions > passes upon test reload mid test execution		`Output` `Screenshots` `Video`

cypress/cypress.cy.js • 3 flaky tests • 5x-driver-electron

View Output Video

Test		Artifacts
... > correctly returns currentRetry		`Output` `Video`
... > correctly returns currentRetry		`Output` `Video`
... > correctly returns currentRetry		`Output` `Video`

The first 5 flaky specs are shown, see all 12 specs in Cypress Cloud.

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

…vantRunSpecsDataSource

lmiller1990 · 2023-03-02T04:21:45Z

I am still going over the code, but this worked well when I tested it. I did real world testing, against an actual production app where I use Cypress for CT and E2E.

I did have to change from and back to the tab to see it update real time - I was just staring at it, expecting it to change - I guess the subscription only triggers if there is a run in progress when you open the page?

Watching it real time was really good - I could start debugging my tests (failed ones) without waiting! Very useful.

warrensplayer · 2023-03-02T13:15:45Z

I did have to change from and back to the tab to see it update real time - I was just staring at it, expecting it to change - I guess the subscription only triggers if there is a run in progress when you open the page?

@lmiller1990 I saw this when developing but then was not able to reproduce it. I will try this scenario again and see if I can make it happen.

warrensplayer · 2023-03-02T13:17:31Z

One thing I'm noticing in initial testing is that there's some flickering back and forth with the navigation badge (seemingly when new data comes in from the cloud?) - I'll see the number of failures, then the badge will flash to "New" for a second, and then back to the number of failures (this time updated since more tests are failing).

@astone123 I have also seen this happen. It seems to only happen once for me and then stop acting that way. If you are able to consistently see it, please try to recall the steps you took to get it to happen.

warrensplayer · 2023-03-02T14:22:08Z

Feedback from acceptance testing with Product and Design

~~add “previous” to banner~~ DONE
~~banner hide delay fix~~ DONE
~~run persistence fix~~ DONE

warrensplayer · 2023-03-02T14:28:34Z

I did have to change from and back to the tab to see it update real time - I was just staring at it, expecting it to change - I guess the subscription only triggers if there is a run in progress when you open the page?

@lmiller1990 I saw this when developing but then was not able to reproduce it. I will try this scenario again and see if I can make it happen.

@lmiller1990 I tested again and I think this is just a matter of waiting for the next outer poll to happen which is every 30 seconds. The "outer" poll is the one from RelevantRunsDataSource that is sending the Git commit shas and determining the appropriate relevant runs. If a run starts recording a split second after one poll, you would have to wait an entire 30 seconds before it polls again. Try your test again and see if that is the case. There are DEBUG logs in the datasources you can turn on to see when it polls.

lmiller1990

This is looking good to me - I'll do a little more testing but the code seems mostly straight forward.

packages/app/src/debug/DebugContainer.cy.tsx

packages/data-context/src/sources/RelevantRunsDataSource.ts

packages/data-context/test/unit/sources/RelevantRunSpecsDataSource.spec.ts

cli/CHANGELOG.md

packages/app/src/debug/DebugContainer.vue

packages/app/src/debug/DebugNewRelevantRunBar.vue

packages/app/src/debug/DebugPageHeader.vue

packages/graphql/test/stubCloudTypes.ts

packages/graphql/src/schemaTypes/objectTypes/gql-RelevantRun.ts

packages/data-context/src/sources/RelevantRunsDataSource.ts

Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com>

mike-plummer

LGTM. Only remaining feedback is that I would like to see a few Percy snapshots added for the new UI behaviors, but that's non-blocking

lmiller1990 · 2023-03-06T23:56:02Z

packages/data-context/test/unit/sources/RelevantRunsDataSource.spec.ts

@@ -147,5 +152,41 @@ describe('RelevantRunsDataSource', () => {
        commitsAhead: 0,
      })
    })
+
+    it('preserves running if switched', async () => {


What does "switching" imply here - when you click on "Change to this run" (or whatever the button says)?

I'm guessing the reason no-one found this edge case in testing was because it was closely related to the timing of when you change runs?

Hope we can find a good way to E2E test this at some point - unit tests do the job, but I imagine it'd be hard to fully grok what this does from a user point of view without an E2E test.

lmiller1990 · 2023-03-06T23:56:22Z

Let's merge it! Asked a question around the latest commit but it's just a question, not a blocker on merging.

warrensplayer added 7 commits February 20, 2023 16:26

update format for duration less than 1 sec

149132b

Move DebugNewRelevantRunBar above header

bb04771

Adding link to switch runs when running

f71ba67

Update DebugPendingRunSplash

5a6276a

New DebugTestingProgress component

03b0c64

Adding DebugTestingProgress to DebugContainer

1587f3a

Adding test for showing failed tests for running

a1db626

warrensplayer added 22 commits February 23, 2023 22:11

Update mutation for moving relevant runs

ed098bd

Fix typescript

9eb299e

Type fix

7bfebae

Fix for Mac OS

38ad010

Watch total tests in relevant run specs data source

08edb5d

Schema change missed earlier

41807b1

Adding transitions to UI

dc89789

Caching gql results between changes to remove UI flicker

fc58e7a

Correcting application of attributes

40e4783

Modify StatsMetadata to incorporate ResultCounts

42942b3

Merge branch 'develop' into stokes/25759_debug_running_builds

dd9e8d4

Adding transition to width

c761907

Update mountFragment and test to match

b21fb72

Adding changelog

ecd8ca2

Update changelog

b627298

Make use of useIntervalFn

d8e15a8

Add interval to timing display in header

96b2433

Rename variable

aa554c8

Fixing GQL warnings from test

0bc767a

Adding entry for Chrome Beta

6ba34cb

Update polling behavior for stopping and fix test for changes in Rele…

bfae5eb

…vantRunSpecsDataSource

Poller updates to tracking start/stop

21befa5

warrensplayer added 3 commits March 2, 2023 10:17

Update link to show previous

be78352

Update runs after a status change

5701bdf

Merge branch 'Develop' into stokes/25759_debug_running_builds

c1aa375

lmiller1990 approved these changes Mar 2, 2023

View reviewed changes

packages/app/src/debug/DebugContainer.cy.tsx Show resolved Hide resolved

packages/data-context/src/sources/RelevantRunsDataSource.ts Outdated Show resolved Hide resolved

packages/data-context/test/unit/sources/RelevantRunSpecsDataSource.spec.ts Show resolved Hide resolved

Update method documentation

a10e303

mike-plummer reviewed Mar 3, 2023

View reviewed changes

warrensplayer and others added 9 commits March 3, 2023 13:34

Update cli/CHANGELOG.md

313dbad

Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com>

Remove console.log

27cec56

Better checks for numbers

d037235

Rename moveToNext to moveToRun

d281af4

Merge branch 'develop' into stokes/25759_debug_running_builds

dfdfc1d

Simplify check

1754865

Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com>

Simplify check

f193904

Fixing document name

2c66474

Merge branch 'develop' into stokes/25759_debug_running_builds

305fd97

warrensplayer requested a review from mike-plummer March 3, 2023 21:28

warrensplayer added 2 commits March 6, 2023 14:04

Update logic for when to move runs

1730dc7

Merge branch 'develop' into stokes/25759_debug_running_builds

33f7cbe

warrensplayer requested a review from lmiller1990 March 6, 2023 19:06

mike-plummer approved these changes Mar 6, 2023

View reviewed changes

lmiller1990 reviewed Mar 6, 2023

View reviewed changes

lmiller1990 approved these changes Mar 6, 2023

View reviewed changes

lmiller1990 merged commit 434e6ee into develop Mar 6, 2023

lmiller1990 deleted the stokes/25759_debug_running_builds branch March 6, 2023 23:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: show running builds in the Debug page #25936

feat: show running builds in the Debug page #25936

warrensplayer commented Feb 24, 2023 •

edited

Loading

cypress bot commented Feb 24, 2023 •

edited

Loading

lmiller1990 commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023

lmiller1990 left a comment

mike-plummer left a comment

lmiller1990 Mar 6, 2023

lmiller1990 commented Mar 6, 2023 •

edited

Loading

feat: show running builds in the Debug page #25936

feat: show running builds in the Debug page #25936

Conversation

warrensplayer commented Feb 24, 2023 • edited Loading

Additional details

Updates

Extra credit

Steps to test

How has the user experience changed?

PR Tasks

cypress bot commented Feb 24, 2023 • edited Loading

25 flaky tests on run #44553 ↗︎

lmiller1990 commented Mar 2, 2023 • edited Loading

warrensplayer commented Mar 2, 2023 • edited Loading

warrensplayer commented Mar 2, 2023 • edited Loading

warrensplayer commented Mar 2, 2023 • edited Loading

warrensplayer commented Mar 2, 2023

lmiller1990 left a comment

Choose a reason for hiding this comment

mike-plummer left a comment

Choose a reason for hiding this comment

lmiller1990 Mar 6, 2023

Choose a reason for hiding this comment

lmiller1990 commented Mar 6, 2023 • edited Loading

warrensplayer commented Feb 24, 2023 •

edited

Loading

cypress bot commented Feb 24, 2023 •

edited

Loading

lmiller1990 commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

warrensplayer commented Mar 2, 2023 •

edited

Loading

lmiller1990 commented Mar 6, 2023 •

edited

Loading