One job ago I worked on a web product that had a pretty good coverage of Selenium tests. Those tests would spawn a Firefox instance and, as such, tended to be slow. On my quad-core Mac Pro they would clock in at a bit over 25 minutes.
A co-worker of mine saw that this was not ideal, time-wise, and set about fixing it. He pulled out Selenium and plugged-in HtmlUnit (via Celerity/Culerity), updated the tests and — BEHOLD! — the test suite ran in four minutes.
Totally. F’ing. Epic.
Fast-forward to now
I’m starting a new web product project and ,naturally, I have some full-stack testing in from the beginning. This time I use capybara-wekbit and all is well. For a while.
Then some warts appear: a case come up where capybara-webkit doesn’t quite render something the way Chrome, Safari,or IE do so although the tests pass, it doesn’t actually work in a real browser. I spend some serious time reworking the code to make it work properly in capybara-webkit as well as the commodity browsers.
All is good again for a time, but then it happened once more. Again, it’s fixed.
The third time this happens, I stop and think about it:
This stuff MUST work in real browsers, and I usually end up testing that by hand. So what’s the point of running tests through this headless browser? Why am I sinking time in to making this work in a browser that, quite literally, NO ONE will actually use?
The revelation was made: the browser being used is part of the stack in full-stack testing.
And, as such, trying to avoid using those browsers directly is, then, avoiding testing part of your stack. A lot has been said that you should develop on the same system as production: same interpreter, same database, same OS. So why should your tests not follow the same logic? Why test using an abstracted browser that, functionally, is never really used by a human?
Unit-testing is a fair point: when you are unit testing JS, a headless browser is probably a good thing to use. It will be faster and (assuming ECMA compliance) just as good as a real browser. But unit-testing is not full-stack testing.
This was a turning point for me. Before that, I saw using Selenium as using a crutch. Now I see that using Selenium is the whole point. You are not just testing your code, you are also testing how your code interacts with the browser. To wit, the browser is in actuality part of your product.
Ok, so, fact: doing Selenium tests with one browser is slow. Doing then with multiple browsers is even slower. But the solution to this problem is not to remove the browser from the equation, is is to make the browser a manageable part of the equation. Farming, parallelism, concurrency, those are all viable ways to make the speed issue more manageable. But removing the browsers from the testing process is not.