Description
Describe the bug
Allure generates individual test case from testNG setup/before methods too if they break. Attached project to reproduce:
bug_reports.zip
To Reproduce
Steps to reproduce the behavior:
- unzip Attached project
- execute
mvn clean test
- execute
cd target
andallure serve
- Check generated report. (I attached screenshot too)
- It contains one additional test "myBeforeClass" which is broken. This confuses testers because 1-the number of tests may be varying in the test class 2-it links a method which may be totally unknown for them (if its located in parent or static class).
- 3-Also It does not add too much value because the same stack can be found linked to the "real" test.
Expected behavior
I would NOT expect to see the setup/before hook's excpetion as independent tests in the report.
I suggest deleting the onConfigurationFailure() method from io/qameta/allure/testng/AllureTestNg.java.
Follow steps to verify:
11. Open attached project
12. comment or delete method onConfigurationFailure() in io/qameta/allure/testng/AllureTestNg.java
13. repeat steps 2,3
14. See that in this case only the real broken test is shown in the list. And we can also find the error starting from this.
BTW: the comment in code also indicates that configuration error should not be separate tests
Screenshots
If applicable, add screenshots to help explain your problem.
5b. if uploaded to allureEE it generated strange orphan tests.
Desktop (please complete the following information):
- OS: [MAC]
- Browser [chrome]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.