Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message for Notifying about Hooks/Non-Tests #4087

Open
thomhurst opened this issue Nov 20, 2024 · 7 comments
Open

Message for Notifying about Hooks/Non-Tests #4087

thomhurst opened this issue Nov 20, 2024 · 7 comments
Labels
Area: MTP Belongs to the Microsoft.Testing.Platform core library Needs: Design Needs research and proposal for how the feature will be implemented.

Comments

@thomhurst
Copy link
Contributor

Summary

A new message that can publish information about hooks/Non-Tests

Background and Motivation

Currently if a hook fails, I have two options:

  • attach the failure to an existing test node
  • create a dummy test node to represent the hook

Those problems with these are, respectively:

  • this implies the test failed which isn't the case, and it also complicates code bases more as the test object must be kept around for longer, and also timings. Do I count hook timings in the test that triggered it? Ideally no, it would have its own object with it's own timing.
  • dummy test nodes are reported as new tests, and would then show under the test counters at the end, and IDEs would show them as runnable tests.

Proposed Feature

A new message to represent hooks would be nice.

It would simplify my logic as I wouldn't have to keep test objects around for longer and delay their result publishing until hooks had finished.

Optionally, supplying a property of test IDs to them could represent what tests they relate to/trigger from. These could be used to drive IDE uis if they wanted to build something to show hook execution.

@Youssef1313 Youssef1313 added the Area: MTP Belongs to the Microsoft.Testing.Platform core library label Feb 24, 2025
@nohwnd
Copy link
Member

nohwnd commented Feb 24, 2025

Do you please have some real-life examples where you would need this so I can better understand the ask? If the hook events are not tied to any test, would they be tied to some place in the test hierarchy, or just to the whole run?

@thomhurst
Copy link
Contributor Author

Things like (in MSTest) ClassInitialize and AssemblyInitialize.

If users define these, and it was to fail I have to either:

  • Create a dummy test node to represent it, which then skews the test count
  • Attach that failure to one of more actual test nodes - when it isn't the tests that failed, it was the setup, so it makes it harder to diagnose the results.

@thomhurst
Copy link
Contributor Author

I guess in a test explorer view, tests are usually grouped by Assembly > Class > Test anyway, so an entry could be added underneath the respected level they're hooking into.

But this new message type shouldn't give the option to be able to run explicitly. Just shown form informative purposes.

@nohwnd
Copy link
Member

nohwnd commented Feb 24, 2025

I am actually not sure if there is some message for this but there should be.

To me those hooks are not super special, it is test explorer that limits us here, and we do have a special mode for MSTest that adds dummy test entries for *Initialize and *Cleanup to be able to represent them in test explorer.

A more MTP oriented solution would be if the IDE understood the relations betwween tests, and then we would simply attach the results to the container that they belong to.

test cleanup error -> test "container"

class cleanup error -> class "container" that contains the tests (or maybe groups of tests and their retries)

assembly cleanup error -> assembly which is parent of all the below,

and even further: single run , run with retries, distributed per computer run, overall run. it scales as far as you need.

This was the intent of having the tree of tests rather than flat list of tests that have some metadata on them. Problem is going further with this idea when users would not be able to see half of the additional data in Test Explorer.

@nohwnd
Copy link
Member

nohwnd commented Feb 24, 2025

To be fair here, test explorer also has its set of challenges, the assembly > class > test group > tests is the default view but far from the only one, so showing the data correctly in the other views is also important for them.

@nohwnd nohwnd added Needs: Design Needs research and proposal for how the feature will be implemented. and removed Needs: Triage 🔍 labels Feb 24, 2025
@nohwnd
Copy link
Member

nohwnd commented Feb 24, 2025

Needs: research on what we have already in place, if anything, and how we could represent this in the MTP, and how we could then show it in TE or VSCode.

The case where we have class or assembly cleanup errors are real asks from customers, but this does not have to be used only for errors.

@Evangelink
Copy link
Member

Nice summary @nohwnd!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: MTP Belongs to the Microsoft.Testing.Platform core library Needs: Design Needs research and proposal for how the feature will be implemented.
Projects
None yet
Development

No branches or pull requests

4 participants