Issue
I have a lot of auto tests. I can run them in any order from Visual Studio 2017 and they work. I'm using WebAppDriver, c# and Selenium.
When we run the tests from Dev-Ops, the tests in the first class work but as soon as it starts running tests from another class, they don't work.
My tests are meant to be independent and able to run in any order but seemingly there is a problem.
I don't want to control the order that the tests run in, they should be independent but I do want to know what has gone wrong.
Each Test Class has a ClassInitialize with a setup that launches the system under test and attaches my driver to it. There is also a Teardown that closes the system under test.
My question is, how can I debug or find out what is going wrong when the tests are run in dev-ops, what's the difference between running the tests from Visual Studio 2017 where they all work and running the tests from dev-ops where they seems to stop working.
So far I have set my tests to run from VS2017 in the same order that dev-ops happens to run them in. I did this by creating a Playlist and editing the playlist file to set the order. It's not my intention to control test order but only for VS2017 to mimic the order that dev-ops runs the tests in. What I have found is that those tests that do run in the chosen order work from VS2017 but don't work in the same order when run from dev-ops.
What is dev-ops doing differently?
Solution
The solution I arrived at was:
in Dev-Ops -> Release Pipeline -> Tasks -> VS Test - Auto Test -> Visual Studio Test (The purple beaker icon), set "Batch Tests = Based on Test Assemblies".
In Visual Studio ensure that each Test Class is in it's own Project and then set the Assembly name of that project to be unique / relevant to that test class (Your Solution may need many projects).
Remove and re-link your auto tests from Visual Studio to the dev-ops test case(s), this is done 1 test at a time.
Then and only then will Dev-Ops be able to run tests from Class A followed by Tests from Class B etc sequentially.
Answered By - Ewan
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.