I’ve been writing Ruby since 2009 and while TDD as a process has long been clear to me, the morass of testing terminology has not. For a recent project I made pretty significant use of Capybara, and through it, Selenium. While it solved some problems I am not sure I could have solved any other way, it created others and on the way shed some light on some murky terminology.
I think mine was a pretty common pathway to working with Capybara, finding it via Rails and Rspec and the need to do an integration test.
The Rspec equivalent of an integration test is the Request Spec, and its often pointed out that its intended use is API testing, which wasn’t what I was doing. What’s held up as the user focused compliment to request specs are feature specs using Capybara.
The sudden appearance of “the feature” as the focus of these specs, and the brief description of “Acceptance test framework for web applications” at the top of the Capybara Github page should be our first signs that things have shifted a little.
This shift in terminology has implications technically, which are not immediately obvious. The intent of Acceptance testing “is to guarantee that a customers requirements have been met and the system is acceptable“. Importantly, since “Acceptance tests are black box system tests”, this means testing the system from the outside via the UI.
Its this “via the UI” part that should stand out, since its a far cry from the other kinds of tests of tests common with Rails. Uncle Bob has said that testing via the UI “is always a bad idea” and I got a taste of why pretty much right away. Lets take a feature spec like this as an example:
it "asks for user email" do
visit '/'
fill_in "user_email", with: "foo@example.com"
click_button "user_submit"
expect(page).to have_content "Thanks for your email!"
end
Notice that suddenly I am stating expectations about the HTML page contents and looking for and manipulating page elements like forms and buttons.
The code for the user_submit button above would typically look like this in most Rails apps:
<%= f.submit "Submit", :class => "btn snazzy" %>
In Rails 3.0.19 that code would use the User class and the input type to create the id attribute automatically. Our click_button 'user_submit' from above finds the element by id and our test passes:
<input class="btn snazzy" id="user_submit" name="commit" type="submit" value="Submit">
In Rails 3.1.12, the same code outputs this:
<input class="btn snazzy" name="commit" type="submit" value="Submit">
In Rails 3.1 they decided to remove the id attribute from the form submit helper so our click_button "user_submit" from the example test above now finds nothing and the test fails.
Rails view helpers exist to abstract away the details of shifting preferences in HTML, form element naming and the tweakery required for cross-browser support. In spite of that I appear to be violating the boundary that abstraction is supposed to provide by making my tests depend on the specific output of the helper.
There are patterns like page objects that can reduce the brittleness but testing via the UI is something that only makes sense when the system under test really is a black box. If you are the developer of said system, it isn’t and you have better options available.
Contrary to the black box assumption of Acceptance tests, Rails integration tests access system internals like cookies, session and the assigns hash as well as asserting specific templates have been rendered. All of that is done via HTTP requests without reliance on clicking UI elements to move between the controllers under test.
Another problem comes from the use of Selenium. By default Capybara uses rack-test as its driver, but adding js: true switches to the javascript driver which defaults to Selenium:
it "asks for user email", js: true do
visit '/'
fill_in "user_email", with: "foo@example.com"
click_button "user_submit"
expect(page).to have_content "Thanks for your email!"
end
The unexpected consequences of this seemingly innocuous option come a month or two later when I try to run my test suite:
Failure/Error: visit '/'
Selenium::WebDriver::Error::WebDriverError:
unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055)
What happened? Well, my package manager has updated my Firefox version to 26.0, and the selenium-webdriver version specified in my Gemfile.lock is 2.35.1.
Yes, with “js: true” I have made that test dependent on a particular version of Firefox, which is living outside of my version control and gets auto-updated by my package manager every six weeks.
While workarounds like bundle update selenium-webdriver or simply skipping tests tagged with js:true using rspec -t ~js:true are available, your default rspec command will always run all the tests. The need to use special options to avoid breakage is unlikely to be remembered/known by future selves/developers, so the solution seems to be keeping some sort of strict separation between your regular test suite and, minimally any test that uses js, or ideally all Acceptance tests. I’m not sure that that might look like yet, but I’m thinking about it.
Acceptance testing differs far more that I initially appreciated from other types of testing, and like most things, when used as intended its pretty fabulous. The unsettling thing in all this was how easy it was to drift into Acceptance testing without explicitly meaning to.
Including the Capybara DSL in integration tests certainly blurs the boundaries between two things that are already pretty easily confused.
Capybara’s ostensible raison d’etre seems to be largely glossed over in most other places as well. Matthew Robbins otherwise great book “Application Testing with Capybara“, is not only conspicuously not called “Acceptance testing with Capybara”, it only mentions the words “Acceptance testing” twice. Not exactly a recipe for a clear-eyed look at the tradeoffs of adopting acceptances testing.
Cabybara is certainly nice to work with, and being able to translate a clients “when I click here it breaks” almost verbatim into a test is amazing. I feel like I now have a better idea of how to enjoy that upside without the downside.
