Backstory

One job ago I worked on a web product that had a pretty good coverage of Selenium tests. Those tests would spawn a Firefox instance and, as such, tended to be slow. On my quad-core Mac Pro they would clock in at a bit over 25 minutes.

Htmlunit_logoA co-worker of mine saw that this was not ideal, time-wise, and set about fixing it. He pulled out Selenium and plugged-in HtmlUnit (via Celerity/Culerity), updated the tests and — BEHOLD! — the test suite ran in four minutes.

Totally. F’ing. Epic.

Fast-forward to now

Webkit_LogoI’m starting a new web product project and ,naturally, I have some full-stack testing in from the beginning. This time I use capybara-wekbit and all is well. For a while.

Then some warts appear: a case come up where capybara-webkit doesn’t quite render something the way Chrome, Safari,or IE do so although the tests pass, it doesn’t actually work in a real browser.  I spend some serious time reworking the code to make it work properly in capybara-webkit as well as the commodity browsers.

All is good again for a time, but then it happened once more. Again, it’s fixed.

The third time this happens, I stop and think about it:

This stuff MUST work in real browsers, and I usually end up testing that by hand. So what’s the point of running tests through this headless browser? Why am I sinking time in to making this work in a browser that, quite literally, NO ONE will actually use?

The revelation was made: the browser being used is part of the stack in full-stack testing.

And, as such, trying to avoid using those browsers directly is, then, avoiding testing part of your stack. A lot has been said that you should develop on the same system as production: same interpreter, same database, same OS. So why should your tests not follow the same logic? Why test using an abstracted browser that, functionally, is never really used by a human?

Full Stack

Unit-testing is a fair point: when you are unit testing JS, a headless browser is probably a good thing to use. It will be faster and (assuming ECMA compliance) just as good as a real browser. But unit-testing is not full-stack testing.

This was a turning point for me. Before that, I saw using Selenium as using a crutch. Now I see that using Selenium is the whole point. You are not just testing your code, you are also testing how your code interacts with the browser. To wit, the browser is in actuality part of your product.

Ok, so, fact: doing Selenium tests with one browser is slow. Doing then with multiple browsers is even slower. But the solution to this problem is not to remove the browser from the equation, is is to make the browser a manageable part of the equation. Farming, parallelism, concurrency, those are all viable ways to make the speed issue more manageable. But removing the browsers from the testing process is not.

  • http://twitter.com/aag1091 aag1091

    fair point …

  • http://twitter.com/mediafinger Andreas Finger

    In my current and former job, we are using a headless browser to test that important workflows do not get broken.

    To test if the app still looks good (and works as expected) in the various browsers each team has it’s own QA guy, that uses test cases to click through the app.

    I would argue that automatic tests can not replace additional manual testing. The time you have to invest to create and maintain a complete automatic test suite for various browsers is too high.

    • xunker

      You make valid points there. In this case, when I talk about “rendering” I mean more of the interaction between the JS and the rendered elements and less about the minute details of the rendered page.

      Automated testing will never completely replace human testing, I agree. But it can augment much of it that doesn’t require a human eye such as the existence or non-existence of elmenets. In this case the rendering issue wasn’t placement, but that an element injected by JS simply wouldn’t show up in the Capybara-webkit frame buffer so you couldn’t do tests against it.

  • http://hron.me/ Gábor Garami

    “This time I use capybara-webkit” – you missed ‘k’

    • xunker

      Thank you, fixed.

  • http://saveyourcall.com/ Eric Anderson

    Has anybody tried using something like BrowserStack? I see they have some sort of testing API. I have wondered if that could be hooked into something like capybara to run tests across the 200+ desktop and mobile browsers they support. Real browsers testing in parallel.

    • xunker

      I have not, but it would be worth looking into. I haven’t checked their API in depth but if you could translate the capybara/selenium stuff into BrowserStack API calls it could be a very viable option.

  • petehawkins

    I don’t think its as cut and dry as you have mentioned here. Full stack tests are to provide value, knowledge to your developers that the code actually works, that nobody introduces a bug, or that the chances of doing so are unlikely. I think if you can get that value from tests in a headless browser that run faster and save you time, then you should use a headless browser. If like you have pointed out, you and your team are just getting burdened by fixing headless only bugs then you should evaluate if the slower browser based tests are worth it and run with that approach.

    I am even of the opinion that *sometimes* full stack tests won’t provide enough value. Rainsberger has a great talk on this: http://www.infoq.com/presentations/integration-tests-scam