Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: with_stderr_(data|contains) has no effect when chained with run_expect_error() #14076

Closed
choznerol opened this issue Jun 15, 2024 · 2 comments · Fixed by #14078
Closed
Labels
A-testing-cargo-itself Area: cargo's tests C-bug Category: bug E-easy Experience: Easy S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review

Comments

@choznerol
Copy link
Contributor

Problem

While working on #14060, I noticed a test case with both with_stderr_contains and run_expect_error would always pass no matter what's passed to with_stderr_contains(...).

  cargo_process("help empty-alias")
        .env("PATH", Path::new(""))
        .with_stderr_contains("THIS RANDOM SENTENCE SHOULD FAIL THIS TEST BUT DIDN'T [..] The subcommand 'empty-alias' wasn't recognized [..]",
        )
        .run_expect_error();

The all 3 usages of run_expect_error in the codebase have the same false positive issue: choznerol@18f4c3f

Steps

  1. Run cargo test --test testsuite help_alias; cargo test --test testsuite config_invalid_empty; cargo test --test testsuite use_mtime_cache_in_cargo_home;. All 3 tests should pass.
  2. Check out this reproduce commit choznerol@18f4c3f on my fork. It just puts random stuff into with_stderr_contains. Rerun step 1 and notice all 3 tests still passed.
  3. (Optional) Same issue even if replacing with_stderr_contains with with_stderr_data. To validate this, you may checkout test: migrate help to snapbox #14060 and run cargo test --test testsuite help_alias.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.79.0-nightly (0637083df 2024-04-02)
release: 1.79.0-nightly
commit-hash: 0637083df5bbdcc951845f0d2eff6999cdb6d30a
commit-date: 2024-04-02
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.3 vendored)
libcurl: 8.4.0 (sys:0.4.72+curl-8.6.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 3.2.1 30 Jan 2024
os: Mac OS 14.4.1 [64-bit]
@choznerol choznerol added C-bug Category: bug S-triage Status: This issue is waiting on initial triage. labels Jun 15, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
@weihanglo
Copy link
Member

I tend to remove run_expect_error entirely and replace with

.with_status(101) // or other exit code
.with_{stderr,stdout}_data(str![....])
.run()

@weihanglo weihanglo added E-easy Experience: Easy A-testing-cargo-itself Area: cargo's tests S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review and removed S-triage Status: This issue is waiting on initial triage. labels Jun 15, 2024
@choznerol
Copy link
Contributor Author

I tend to remove run_expect_error entirely and replace with

.with_status(101) // or other exit code
.with_{stderr,stdout}_data(str![....])
.run()

Thanks! Let me make a PR to fix the 3 tests and remove run_expect_error().

choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
The `with_stderr_contains()` (as well as the new `with_stderr_data()` too, see rust-lang#14060) has no effect when using with `run_expect_error()`.
choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 15, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 16, 2024
The `with_stderr_contains()` (as well as the new `with_stderr_data()` too, see rust-lang#14060) has no effect when using with `run_expect_error()`.
choznerol added a commit to choznerol/cargo that referenced this issue Jun 16, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 16, 2024
choznerol added a commit to choznerol/cargo that referenced this issue Jun 16, 2024
@bors bors closed this as completed in 5586a47 Jun 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testing-cargo-itself Area: cargo's tests C-bug Category: bug E-easy Experience: Easy S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants