-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Message for Notifying about Hooks/Non-Tests #4087
Comments
Do you please have some real-life examples where you would need this so I can better understand the ask? If the hook events are not tied to any test, would they be tied to some place in the test hierarchy, or just to the whole run? |
Things like (in MSTest) ClassInitialize and AssemblyInitialize. If users define these, and it was to fail I have to either:
|
I guess in a test explorer view, tests are usually grouped by Assembly > Class > Test anyway, so an entry could be added underneath the respected level they're hooking into. But this new message type shouldn't give the option to be able to run explicitly. Just shown form informative purposes. |
I am actually not sure if there is some message for this but there should be. To me those hooks are not super special, it is test explorer that limits us here, and we do have a special mode for MSTest that adds dummy test entries for A more MTP oriented solution would be if the IDE understood the relations betwween tests, and then we would simply attach the results to the container that they belong to. test cleanup error -> test "container" class cleanup error -> class "container" that contains the tests (or maybe groups of tests and their retries) assembly cleanup error -> assembly which is parent of all the below, and even further: single run , run with retries, distributed per computer run, overall run. it scales as far as you need. This was the intent of having the tree of tests rather than flat list of tests that have some metadata on them. Problem is going further with this idea when users would not be able to see half of the additional data in Test Explorer. |
To be fair here, test explorer also has its set of challenges, the assembly > class > test group > tests is the default view but far from the only one, so showing the data correctly in the other views is also important for them. |
Needs: research on what we have already in place, if anything, and how we could represent this in the MTP, and how we could then show it in TE or VSCode. The case where we have class or assembly cleanup errors are real asks from customers, but this does not have to be used only for errors. |
Nice summary @nohwnd! |
Summary
A new message that can publish information about hooks/Non-Tests
Background and Motivation
Currently if a hook fails, I have two options:
Those problems with these are, respectively:
Proposed Feature
A new message to represent hooks would be nice.
It would simplify my logic as I wouldn't have to keep test objects around for longer and delay their result publishing until hooks had finished.
Optionally, supplying a property of test IDs to them could represent what tests they relate to/trigger from. These could be used to drive IDE uis if they wanted to build something to show hook execution.
The text was updated successfully, but these errors were encountered: