Test Fixes to Run Xcode 15 in CI #22270

jostnes · 2023-12-22T05:57:20Z

Description

This is the "cleaned up" version of @mokagio's PR #21921, with only the final working commit and the many experimentation commits from the original PR omitted (see original PR for more description about the changes)

The branch was tested on WPiOS-macv2-test with the UI test steps passing on every build. There are still some flaky tests, but automated retries currently handle those, and it would be better to handle them separately if they become too flaky. The current run time for UI test is still within 30 minutes (same as before).

The UI test fixes that were made:

Somehow stats test started failing/became flakier with this change. I updated the stats test and mocks to be more dynamic; the year is no longer hard coded to 2019 but follows the current year. Flakiness is still seen when the screen is not updated to use the values presented in the mocks; I couldn't reproduce this on a real device, which is why this could be a mock-related issue.
On reply notification, another flakiness is seen where the tap does not work (based on screenshots, the tap did happen, but the action that should have happened after the tap did not. Also, it is not reproducible on a real device). I renamed tapUntilCondition to waitForElementAndTap and updated it to cover more conditions to make it more robust. The change works for now.
To workaround the simulator bug where we can't turn off Save Password from settings, app.dismissSavePasswordPrompt() was readded after password screen.

PR submission checklist:

I have completed the Regression Notes.
I have considered adding unit tests for my changes.
I have considered adding accessibility improvements for my changes.
I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

UI Changes testing checklist: Not a UI PR

tiagomar · 2024-01-02T19:02:19Z

The executeWithRetries(disableAutoFillPasswords) in testBundleWillStart from TestObserver is still running. Is the idea still to skip it for now)?

tiagomar

Thanks for working on this, @jostnes ! 🙇

I've run all tests locally on Xcode 15.1 and all tests passed on iPhone and iPad. I just left a couple comments before approving this PR.

tiagomar · 2024-01-02T17:41:11Z

WordPress/UITestsFoundation/Globals.swift

@@ -62,20 +62,29 @@ public func waitAndTap( _ element: XCUIElement, maxRetries: Int = 20) {
    }
 }

-public func tapUntilCondition(element: XCUIElement, condition: Bool, description: String, maxRetries: Int = 10) {
+public func waitForElementAndTap(_ tapElement: XCUIElement, untilConditionOn conditionElement: XCUIElement, condition: String, errorMessage: String, maxRetries: Int = 10) {


I'm not sure waitForElementAndTap describes exactly what the function does? The function doesn't actually wait for the tapElement before tapping on it.

WDYT about tapUntil(_ tapElement: XCUIElement, conditionOn conditionElement: XCUIElement, condition: String, errorMessage: String, maxRetries: Int = 10)?

@tiagomar 's observation resonates with me. As a matter of fact, I opened #22322 to address it (with an attempt to improve upon it).

Having said that, I don't see this as a blocker to merge given the benefit that merging this PR to add Xcode 15.1 / iOS 17.2 support would bring to the dev teams.

I agree, @mokagio, it makes sense to move on with this PR and address the naming in another one. 👍

"Typed" in that it expects an `enum` instead of a `String` to define the expected element state.

It became unused in the previous commit, b442842

mokagio

@jostnes @tiagomar I'd love to see this merged sooner rather than later, even if it means not addressing the naming comment straightaway and instead follow up later. Rationale: the sooner we officially support Xcode 15.1 and iOS 17+ in CI, the better.

mokagio · 2024-01-03T03:41:01Z

WordPress/UITestsFoundation/Globals.swift

@@ -62,20 +62,29 @@ public func waitAndTap( _ element: XCUIElement, maxRetries: Int = 20) {
    }
 }

-public func tapUntilCondition(element: XCUIElement, condition: Bool, description: String, maxRetries: Int = 10) {
+public func waitForElementAndTap(_ tapElement: XCUIElement, untilConditionOn conditionElement: XCUIElement, condition: String, errorMessage: String, maxRetries: Int = 10) {


@tiagomar 's observation resonates with me. As a matter of fact, I opened #22322 to address it (with an attempt to improve upon it).

Having said that, I don't see this as a blocker to merge given the benefit that merging this PR to add Xcode 15.1 / iOS 17.2 support would bring to the dev teams.

staskus · 2024-01-03T06:49:21Z

Note: I have widget API compliance for Xcode 15 + iOS 17 lined up (#21705). So it's not a blocker if you notice anything wrong related to widgets.

To follow up via #22323

tiagomar

Approving this PR now as the naming issue will be addressed in #22322.

🤦‍♂️ I didn't notice the `!` at the start of the check when porting it over.

jostnes · 2024-01-04T02:05:18Z

There's a unit test failing in the PR. I'll update with the latest trunk again to see if it solves the issue and to be able to merge this.

tiagomar · 2024-01-04T02:39:01Z

There's a unit test failing in the PR. I'll update with the latest trunk again to see if it solves the issue and to be able to merge this.

Yeah, it has been taken care of in 7365c10. #22322 is ready to be merged into this branch.

…#22322)

jostnes · 2024-01-04T04:12:18Z

@mokagio the build is good on the iOS 17+ pipeline in CI but some steps are failing on the pipeline checks linked to this PR because of Unable to find remote image: xcode-15.1 (which i guess is expected)

should the GitHub checks be updated for this? at least one of the failing step is marked as required so we can't merge this yet.

mokagio · 2024-01-04T08:11:38Z

@jostnes yes sorry, I should have left a note that before being able to merge this we need to move WordPress-iOS on Buildkite to the Apple Silicon cluster.

I have a PR for it that I'll mention internally.

guarani · 2024-01-04T21:56:05Z

Note: I have widget API compliance for Xcode 15 + iOS 17 lined up (#21705). So it's not a blocker if you notice anything wrong related to widgets.

👋 Will the 24.0 beta be built using Xcode 15? If so, I think #21705 needs to be merged before Monday to avoid widgets breaking in the beta builds. Apologies, it's been waiting for my review for a while now, I'll get to it ASAP.

wpmobilebot · 2024-01-08T03:46:32Z

📲 You can test the changes from this Pull Request in WordPress Alpha by scanning the QR code below to install the corresponding build.

	App Name	WordPress Alpha
	Configuration	Release-Alpha
	Build Number	pr22270-5bafb56
	Version	23.9
	Bundle ID	org.wordpress.alpha
	Commit	`5bafb56`
	App Center Build	WPiOS - One-Offs #8342

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

wpmobilebot · 2024-01-08T03:46:48Z

📲 You can test the changes from this Pull Request in Jetpack Alpha by scanning the QR code below to install the corresponding build.

	App Name	Jetpack Alpha
	Configuration	Release-Alpha
	Build Number	pr22270-5bafb56
	Version	23.9
	Bundle ID	com.jetpack.alpha
	Commit	`5bafb56`
	App Center Build	jetpack-installable-builds #7365

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

This reverts commit 39cb20f.

peril-wordpress-mobile · 2024-01-08T03:58:56Z

	Warnings
⚠️	This PR is assigned to a milestone which is closing in less than 4 days Please, make sure to get it merged by then or assign it to a later expiring milestone

Generated by 🚫 dangerJS

wpmobilebot · 2024-01-08T04:04:21Z

	1 Warning
⚠️	This PR is assigned to the milestone 24.0. This milestone is due in less than 4 days. Please make sure to get it merged by then or assign it to a milestone with a later deadline.

Generated by 🚫 Danger

…ring This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

…-widgets-xcode-15-ios-17 This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270 --- There were conflicts because the Gutenberg version move forward on `trunk` and this branch. I resolved by keeping Gutenberg to the version here because that's the point of this branch, but given the time delta it's possible the build will fail.

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

jostnes added 3 commits December 19, 2023 13:38

add working changes from mokagio/fix-code-15-tests branch

49aedbb

increase timeouts on verifyStatsLoaded and verifyChartLoaded

8be833b

Merge branch 'trunk' into ui-test-fixes-ios17.2

11aa3d4

jostnes requested a review from a team as a code owner December 22, 2023 05:57

Merge branch 'trunk' into ui-test-fixes-ios17.2

6ba9000

jostnes mentioned this pull request Dec 28, 2023

Stats: Data shown for incorrect period after navigating back and returning #21099

Closed

tiagomar requested changes Jan 2, 2024

View reviewed changes

mokagio added this to the 24.0 milestone Jan 3, 2024

mokagio added the Tooling Build, Release, and Validation Tools label Jan 3, 2024

mokagio mentioned this pull request Jan 3, 2024

Run Xcode 15 in CI and skip or disable tests that fail #21921

Closed

4 tasks

Reimplement waitForElementAndTap... as a typed XCUIElement method

b442842

"Typed" in that it expects an `enum` instead of a `String` to define the expected element state.

mokagio mentioned this pull request Jan 3, 2024

Reimplement waitForElementAndTap... as a typed XCUIElement method #22322

Merged

4 tasks

Remove unused waitForElementAndTap method.

05597db

It became unused in the previous commit, b442842

mokagio approved these changes Jan 3, 2024

View reviewed changes

Add a check for the "Not Now" button existence

7f86de8

Allow a date-bound unit test to fail

7365c10

To follow up via #22323

tiagomar self-requested a review January 3, 2024 12:30

tiagomar approved these changes Jan 3, 2024

View reviewed changes

mokagio and others added 3 commits January 4, 2024 09:32

Fix logic to evaluate .dismissed condition

de82f79

🤦‍♂️ I didn't notice the `!` at the start of the check when porting it over.

Use consistent indentation in XCUIElement+TapUntil

d2cba4d

Merge branch 'trunk' into ui-test-fixes-ios17.2

a1ebec0

wordpress-mobile deleted a comment from jacob121532 Jan 4, 2024

Reimplement waitForElementAndTap... as a typed XCUIElement method (…

cdc34d1

…#22322)

mokagio enabled auto-merge January 8, 2024 02:19

Work around unit test failing because of 2023 vs 2024 issue

39cb20f

mokagio and others added 3 commits January 8, 2024 14:52

Address RuboCop violation

a6b9305

Revert "Work around unit test failing because of 2023 vs 2024 issue"

faa1d72

This reverts commit 39cb20f.

Tests: Fix failure in BloggingPromptsServiceTests

5bafb56

mokagio merged commit 7b65cbc into trunk Jan 8, 2024

mokagio deleted the ui-test-fixes-ios17.2 branch January 8, 2024 04:33

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/22315-backend-driven-dynamic-cards-filte…

8abe2f6

…ring This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into fix-issue-22283-page-list-hierachy

eac5db6

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/remove-unused-code-p6

5f78773

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/remove-confirmation-on-trash

fcaf9a2

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into fix/crash-sync-media

0f9bc78

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/improve-media-item-preview

86dbb14

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into feat/update-image-block-upload-visuals

79e311c

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/add-prefetching-to-media-gallery

343ed2c

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/remove-deprecated-document-picker-api

b120149

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into fix/please-adopt-container-background-api-for…

1e80d49

…-widgets-xcode-15-ios-17 This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/simpler-pch

b9a4ca4

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/21501-this-week-chart-widget

477d1bc

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into fix/21867-recommended-tags-layout-issue

2255b4e

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

This was referenced Jan 8, 2024

In App Feedback Prompt Groundwork #22050

Merged

In-App Feedback Prompt UI #22052

Merged

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into ui-test-fix-free-to-paid-card-navigation

d4ae2d7

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into iangmaia/move-danger-to-gha

23efeda

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

mokagio added a commit that referenced this pull request Jan 8, 2024

Merge remote trunk into task/remove-migrations

171b38f

This is necessary to get up to date with a CI configuration change that requires running on Xcode 15.1. See #22270

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Fixes to Run Xcode 15 in CI #22270

Test Fixes to Run Xcode 15 in CI #22270

jostnes commented Dec 22, 2023

tiagomar commented Jan 2, 2024

tiagomar left a comment

tiagomar Jan 2, 2024

mokagio Jan 3, 2024

tiagomar Jan 3, 2024

mokagio left a comment

mokagio Jan 3, 2024

staskus commented Jan 3, 2024

tiagomar left a comment

jostnes commented Jan 4, 2024

tiagomar commented Jan 4, 2024

jostnes commented Jan 4, 2024

mokagio commented Jan 4, 2024

guarani commented Jan 4, 2024

wpmobilebot commented Jan 8, 2024 •

edited

Loading

wpmobilebot commented Jan 8, 2024 •

edited

Loading

peril-wordpress-mobile bot commented Jan 8, 2024

wpmobilebot commented Jan 8, 2024

Test Fixes to Run Xcode 15 in CI #22270

Test Fixes to Run Xcode 15 in CI #22270

Conversation

jostnes commented Dec 22, 2023

Description

tiagomar commented Jan 2, 2024

tiagomar left a comment

Choose a reason for hiding this comment

tiagomar Jan 2, 2024

Choose a reason for hiding this comment

mokagio Jan 3, 2024

Choose a reason for hiding this comment

tiagomar Jan 3, 2024

Choose a reason for hiding this comment

mokagio left a comment

Choose a reason for hiding this comment

mokagio Jan 3, 2024

Choose a reason for hiding this comment

staskus commented Jan 3, 2024

tiagomar left a comment

Choose a reason for hiding this comment

jostnes commented Jan 4, 2024

tiagomar commented Jan 4, 2024

jostnes commented Jan 4, 2024

mokagio commented Jan 4, 2024

guarani commented Jan 4, 2024

wpmobilebot commented Jan 8, 2024 • edited Loading

wpmobilebot commented Jan 8, 2024 • edited Loading

peril-wordpress-mobile bot commented Jan 8, 2024

wpmobilebot commented Jan 8, 2024

wpmobilebot commented Jan 8, 2024 •

edited

Loading

wpmobilebot commented Jan 8, 2024 •

edited

Loading