One of my current clients is really embracing all the MS clones of great open source tools like MSTest, Team Build, Entity Framework, etc. Although I love the open source tools like nUnit, Team City (ok not so open source but still free), and nHibernate, I thought I would try MS Test out even though I have not heard many good things about them.
First I will say something nice: MSTest has a nice code coverage suite. I really like that it is integrated with visual studio and can even highlight the areas that are covered and not covered which is the ONE feature of MSTest that I like.
Other than that the code coverage I think MS test is a complete miss when it comes to development for the following reasons:
- Only being able to add tests to a test project. I like my unit test classes to sit in the same project as the code I am testing as I find it easier to navigate the code, easy to spot if I did not test a class (As there is no [ClassName]Tests.cs file next to it. It appears that with MS Test I can only add tests to a test project
- VSMDI File. When a test project is created it adds a .vsmdi and .testrunconfig file to the solution. Why these are needed at the solution level (or at all), I have no idea. Adding these items makes me have several spots to go to for changing settings instead of just one
- Poor IDE integration. In VS I now have a test view tab, a test results tab, and a test runs tab docked into the editor (plus a code coverage tab but I can agree with this being a seprate tab). All I need is one area where I can run the tests and see the results. Just a heads up but SIMPLE THINGS ARE EASY TO USE!
- Notification about having run more than 25 test runs. Apparently MS Test keeps 25 copies of the old tests around and when you try to run your unit tests for the 26th time you get a warning about this which is annoying. In 99% of situations you will not need to go back to the first time you ran tests so why bother even telling me?
- Release/Debug Suprises: I had a set of tests constantly failing in the MSTest runner yet constantly passing in my Resharper runner. Also the code coverage went screwy (the code coverage was not accurate and line highlighting did not match up). After a few hours I discovered that it was because the app was set to compile to a release but MS test just look at the debug DLL’s with no warning that it is doing so. I agree that the same thing would happen with nUnit but nUnit is not integrated into the UI and is therefore not expected to be able to pick up on these things MSTest is supposed to be this integrated wonder that should know when the tests are compiled in release to maybe use the release code instead of the stale debug code (or at least warn me)
- Hard to Navigate. When a test fails you can see some of the description of the error message but if you want further details you need to double click the item which oepns a new editor tab which explains the error a bit more. I find having to manage these tabs an annoyance and much perfer the nUnit or Resharper test runner approaches myself.
- Hard to just run one set of tests. I have two test projects one for unit tests and one for functional tests (items that hit the database and other costly resources). It seems hard to just run tests for one of the projects. It is also hard to run tests for just one namespace. Again these things are easy with other unit testing frameworks.
- The context menus are confusing. Right clicking on the test project and selecting “add” yeilds “New Test”, “Unit Test”, and “Ordered Test”.
New Test: Gives you the option of creating a new test, an ordered test, or use the test wizard
Unit Test: launches the test wizard
Orderd Test: creates a new ordered test
I find the multiple routes a bit confusing (not to mention how it adds 3 elements to my already crowded context menu when I only need one)
Now these are just the issues I have had with it. A quick stop on a search engine brings up many other issues but so far I have not experienced them. I find that MSTest is not as clean, fast, or simple as other unit testing frameworks out there which makes me not want to use it on another project.