Mastodon

Tag: rspec

On working with Capybara

I’ve been writing Ruby since 2009 and while TDD as a process has long been clear to me, the morass of testing terminology has not. For a recent project I made pretty significant use of Capybara, and through it, Selenium. While it solved some problems I am not sure I could have solved any other way, it created others and on the way shed some light on some murky terminology.

I think mine was a pretty common pathway to working with Capybara, finding it via Rails and Rspec and the need to do an integration test.

The Rspec equivalent of an integration test is the Request Spec, and its often pointed out that its intended use is API testing, which wasn’t what I was doing. What’s held up as the user focused compliment to request specs are feature specs using Capybara.

The sudden appearance of “the feature” as the focus of these specs, and the brief description of “Acceptance test framework for web applications” at the top of the Capybara Github page should be our first signs that things have shifted a little.

This shift in terminology has implications technically, which are not immediately obvious. The intent of Acceptance testing “is to guarantee that a customers requirements have been met and the system is acceptable“. Importantly, since “Acceptance tests are black box system tests”, this means testing the system from the outside via the UI.

Its this “via the UI” part that should stand out, since its a far cry from the other kinds of tests of tests common with Rails. Uncle Bob has said that testing via the UI “is always a bad idea” and I got a taste of why pretty much right away. Lets take a feature spec like this as an example:

    it "asks for user email" do
      visit '/'
      fill_in "user_email", with: "foo@example.com"
      click_button "user_submit"
      expect(page).to have_content "Thanks for your email!"
    end

Notice that suddenly I am stating expectations about the HTML page contents and looking for and manipulating page elements like forms and buttons.
The code for the user_submit button above would typically look like this in most Rails apps:

<%= f.submit "Submit", :class => "btn snazzy" %>

In Rails 3.0.19 that code would use the User class and the input type to create the id attribute automatically. Our click_button 'user_submit' from above finds the element by id and our test passes:

<input class="btn snazzy" id="user_submit" name="commit" type="submit" value="Submit">

In Rails 3.1.12, the same code outputs this:

<input class="btn snazzy" name="commit" type="submit" value="Submit">

In Rails 3.1 they decided to remove the id attribute from the form submit helper so our click_button "user_submit" from the example test above now finds nothing and the test fails.

Rails view helpers exist to abstract away the details of shifting preferences in HTML, form element naming and the tweakery required for cross-browser support. In spite of that I appear to be violating the boundary that abstraction is supposed to provide by making my tests depend on the specific output of the helper.

There are patterns like page objects that can reduce the brittleness but testing via the UI is something that only makes sense when the system under test really is a black box. If you are the developer of said system, it isn’t and you have better options available.

Contrary to the black box assumption of Acceptance tests, Rails integration tests access system internals like cookies, session and the assigns hash as well as asserting specific templates have been rendered. All of that is done via HTTP requests without reliance on clicking UI elements to move between the controllers under test.

Another problem comes from the use of Selenium. By default Capybara uses rack-test as its driver, but adding js: true switches to the javascript driver which defaults to Selenium:

    it "asks for user email", js: true do
      visit '/'
      fill_in "user_email", with: "foo@example.com"
      click_button "user_submit"
      expect(page).to have_content "Thanks for your email!"
    end

The unexpected consequences of this seemingly innocuous option come a month or two later when I try to run my test suite:

     Failure/Error: visit '/'
     Selenium::WebDriver::Error::WebDriverError:
       unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055)

What happened? Well, my package manager has updated my Firefox version to 26.0, and the selenium-webdriver version specified in my Gemfile.lock is 2.35.1.

Yes, with “js: true” I have made that test dependent on a particular version of Firefox, which is living outside of my version control and gets auto-updated by my package manager every six weeks.

While workarounds like bundle update selenium-webdriver or simply skipping tests tagged with js:true using rspec -t ~js:true are available, your default rspec command will always run all the tests. The need to use special options to avoid breakage is unlikely to be remembered/known by future selves/developers, so the solution seems to be keeping some sort of strict separation between your regular test suite and, minimally any test that uses js, or ideally all Acceptance tests. I’m not sure that that might look like yet, but I’m thinking about it.

Acceptance testing differs far more that I initially appreciated from other types of testing, and like most things, when used as intended its pretty fabulous. The unsettling thing in all this was how easy it was to drift into Acceptance testing without explicitly meaning to.

Including the Capybara DSL in integration tests certainly blurs the boundaries between two things that are already pretty easily confused.

Capybara’s ostensible raison d’etre seems to be largely glossed over in most other places as well. Matthew Robbins otherwise great book “Application Testing with Capybara“, is not only conspicuously not called “Acceptance testing with Capybara”, it only mentions the words “Acceptance testing” twice. Not exactly a recipe for a clear-eyed look at the tradeoffs of adopting acceptances testing.

Cabybara is certainly nice to work with, and being able to translate a clients “when I click here it breaks” almost verbatim into a test is amazing. I feel like I now have a better idea of how to enjoy that upside without the downside.

Ruby Redos: Rspec

I have always found Rspec syntax fascinating. There is something incredible about being able to type readable English sentences and have them do exactly what you think they will from reading them.
I wanted to take a stab at recreating something like Rspec and it was a great exercise. Test::Unit has always felt foreign and this little experiment was a great reminder of why: It ignores Ruby’s blocks.

Test::Unit works by gathering all the decendents of Test::Unit::TestCase and calling every method that starts with “test_”:

class TestMyClass < Test::Unit::TestCase
  def test_my_method1
    ... 
  end 

  def test_my_method2
    ... 
  end 
end

This approach, to my eyes, is a relative of the command pattern, which is essentially a workaround for the lack of language support for blocks. It does not make much sense to use the command pattern in Ruby but you still see it pop up all over the place.

Rspec syntax, on the other hand, is really a just a bunch of Ruby blocks:

[14] pry(main)> load 'test.rb'
=> true
[15] pry(main)> describe "Thing" do                                                                       
[15] pry(main)*   it "is nil" do                                                                          
[15] pry(main)*     expect(nil).to be_nil                                                                 
[15] pry(main)*   end  
[15] pry(main)* end  
A Thing:
is nil
-----

You will notice that describe is defined on the main object and that everything else I have put in a TestContext class to house the methods that I am expecting to be called in the blocks. Then I am using instance eval to execute the block as though it were part of the TestContext class. One other thing to point out is that calling Proc.new will grab any block that has been passed to the method:

require 'benchmark'

def describe subject
  puts "A #{subject}:"
  block = Proc.new
  Benchmark.measure do
  TestContext.new.run(block)
  end.real
end


class TestContext

  def initialize
    @examples = {}
  end

  def run block
    instance_eval &block
    @examples.each_pair do |desc, code|
      if code.call
        puts "  \033[32m#{desc}\033[0m\n"
      else
        puts "  \033[31m#{desc}\033[0m\n"
      end
      puts "-----"
    end
  end

  def it its_description
    @examples[its_description] = Proc.new
  end

  def expect thing
    Assertion.new(thing)
  end

  def be_nil
    NilMatcher.new
  end

end

class NilMatcher

  def match subject
    subject.nil?
  end

end

class Assertion

  def initialize subject
    @subject = subject
  end

  def to what
    what.match(@subject)
  end

end

I like the fact that the emphasis is on how a few methods read when chained (describe, it, expect, to, be_nil) and using those as a thin layer around blocks. This is some of the core stuff that that makes Ruby great to work with and, in my opinion, a great example of what Matz is talking about when he says “Languages can be weapons against complexity”.
He has given us blocks in the language so things as common as “wanting to execute code later” is actually part of the language. No design pattern required. The flexibility of the language also means we can write code with a very small distance between what is said and what is meant, further reducing the complexity. Both of those things are on display in Rspec.

Factory not registered

I have been working on adding tests to an existing Rails app for one of my clients. The app was written by some well intentioned person who included tonnes of testing libraries but just never got around to writing tests.

group :development, :test do
  gem 'capybara'
  gem 'cucumber'
  gem 'cucumber-rails', :require => false
  gem 'database_cleaner'
  gem 'factory_girl'
  gem 'factory_girl_rails'
  gem 'rspec-core'
  gem 'rspec-expectations'
  gem 'rspec-mocks'
  gem 'rspec'
  gem 'rspec-rails'
  gem 'ZenTest'
end

As I started adding specs I needed to create factories, and when I started to do that I started to get errors. First “Factory already registered” errors and then “Factory not registered” errors. Very annoying.
The root of the problem seems to be that factory_girl_rails and factory_girl both look for factories when loaded. The solution for me was to remove factory_girl and let factory_girl_rails do the right thing.
This triggered a major purge of unneeded gems which left me with a Gemfile that looks like this:

group :development, :test do
  gem 'factory_girl_rails'
  gem 'rspec-rails'
  gem 'capybara'
end

It also left me with passing tests and no further errors.

I think the take away for me here is to let bundler/rubygems dependency management system do the right thing. Over-specifying stuff is a recipe for trouble. Over-specifying stuff in the Gemfile might look like explicitly including factory_girl when it will obviously be a dependency of factory_girl_rails. I’ve also seen problems arise from habitually including version numbers in the Gemfile, which makes things so specific that Bundler no longer has the latitude to figure out something that works.

In any case its a good reminder that less is more.

The difference between Feature Specs and Request Specs

Rspec has two different types of tests that are very similar to integration tests; request specs and feature specs. The difference between them has never been particularly clear to me and seems to be a point of confusion for many others as well.

Feature specs and Request specs are both are part of rspec-rails and are built on a similar foundation.

Request specs are built on top of Rails integration tests, which offer an API with methods that mirror what you find in HTTP: “get”, “put”, “post”, etc.
It gives you a “response” object for you to make your assertions against.

Feature specs are built on top of Capybara. The goal of Capybara is to offer an API that “simulates user behaviour”. To that end their API employs language that is not far from what you would use when directing someone to use a website over the phone:

visit '/sessions/new'
fill_in 'Login', :with => 'user@example.com'
fill_in 'Password', :with => 'password'
click_link 'Sign in'
page.should have_content 'Success'

Under the covers “visit” is just calling Rack::Test methods but there is no low-level, HTTP oriented language (request, response, get, put, etc…) used in the API itself. Capybara is consciously omitting them as Jonas Nicklas (Capybara’s author) explains on his blog.

Feature specs exist to provide a place for the type of high level testing that Capybara enables. The Relishapp site describes them as “high-level tests meant to exercise slices of functionality through an application. They should drive the application only via its external interface, usually web pages.” This unfortunately jargon-laden description, really only made sense after some digging.

Which brings us to Request specs. These are for those moments when the details of HTTP are exactly what you care about; like when you are testing an API. 

get "/todays-forcast"
expect(response.body).to have_content "sunny"

Here the details of HTTP are front and centre, because in an API they matter. This is why request specs are referred to as ” a thin wrapper around Rails’ integration tests”, the focus is on providing just some Rspec sugar to use with the existing HTTP oriented methods.

The confusion around these things seem to stem from the early days of the project. When Capybara was first created it was intended to be used within request specs. That created a bunch of confusion because of the competing syntaxes ( get/visit, response/page) which resulted in the creation of feature specs to give a proper home to this type of testing.

Unfortunately people are still following old instructions and this usage is still common which muddies the water when trying to understand which is which. Hopefully this adds a little clarity.