The maze book for programmers!
mazesforprogrammers.com

Algorithms, circle mazes, hex grids, masking, weaving, braiding, 3D and 4D grids, spheres, and more!

DRM-Free Ebook

The Buckblog

assorted ramblings by Jamis Buck

How would YOU do it?

23 January 2007 — 1-minute read

I’ve been working on rewiring parts of Capistrano, and I’m making some good progress in a few different areas. One thing that has continued to defy me is actually unit testing Capistrano. I openly admit the existing unit tests for Capistrano are remarkably lame, but I’m at a loss as to how to improve them.

So, I’m opening the floor on this one. How would you do it? Specifically, allow me to point you to the gateway implementation, which uses Net::SSH and sockets, and which has proven particularly tricky for me to see my way through to unit testing. Other areas that I would appreciate suggestions are: what is a good way to go about testing the CLI (command-line interface) module? What would be a technique for actually testing the standard tasks? And how would you test the file transfer abstraction and the capistrano shell?

I know from experience that code which is hard to test is usually poorly designed, so I’m prepared to hear comments like “you’ve misdesigned this thing, you should have done X, Y, and Z.” Of course, the gentler you can break it to me, the happier I’ll be. ;)

So, tell me, how would you do it?

Reader Comments

Testing network code is never easy :( and these examples you give seem especially hard.

I usually use flexmock for these kinds of problems, and mock out as high a level as I can, to focus my tests on the specific method.

For instance in the file transfer abstraction, I’d flexmock sftp, and have it return different results for testing those edge cases.

I found flexmock to be the most flexible of the mocking packages, although it takes a while to get the hang of it, but well worth it.

Thanks for the tip, Jim, I don’t feel quite so stupid knowing that the examples given are “especially hard” to test. I’ll see about using flexmock more for some of those.

For testing the commandline components, I’d use expect. Not Ruby, but it does do a good job of testing commandline tooling.

Mocha (http://mocha.rubyforge.org/) might be useful?

Just submit it to that Rails Way site. Those guys are smart, they’ll help you.

I was going to say – use Mocks – but I see someone beat me to it.

Seriously, though, I discovered something interesting in some of my fellow programmers. They don’t use mocks because they say mocks are too hard to build. But that’s just a cover for the real problem, which is that they don’t understand the thing needing to be “mocked” well enough to mock it… which sort of explains why the unit tests suck.

I agree with Lori, mocking is an acquired art, and it isn’t always obvious how to mock an object or indeed which objects to mock. Also injection of mock objects sometimes requires minor refactoring of the code under test, which I think is quite acceptable practice as good unit tests have such obvious advantages.

I will admit that I am not as familiar with mocking and stubbing as I would like to be—I’ve done it some (with both Net::SSH and Capistrano, even), but I’d like to understand it better. Can anyone recommend any resources for becoming better acquainted? Trial and error are demanding teachers, any cliff notes for that course are welcome. :)

Start with Martin Fowler’s essay http://www.martinfowler.com/articles/mocksArentStubs.html

Then dig around at http://www.mockobjects.com/

I had to do something similar with Net::HTTP. Basically tesing how the app would interact with a Web server that could return a page, 404, redirect, etc.

After digging through Net::HTTP I found what I was looking for. All the API calls (start, get, get_response) come down to one internal method that makes the actual HTTP request. Ruby has open classes, so it was just a matter of redefining that one method. Instead of doing HTTP calls, it would execute a block of my choosing, simulating a server.

I could then script how the server behaves by writing something like this in my test case:

  1. Setup Net::HTTP to simulate a server. Net::HTTP.in_response_to “http://example.com/index.html” do return [200, “Hello”] end
  2. Actual test comes here …

That worked well for simple case. But say you want to test that the library will make one request, server returns a redirect, the library would then make another request to the new URL, get a page and be done. If you think about it for a second, that requires a lot of state management and assertions. It quickly becomes unreadable.

So instead, I changed Net::HTTP to call a mock object, making different calls depending on the request URI. I could then write the expected behavior:

  1. Decide how the server will behave. mock.once.expect(”/”).response { redirect_to “index.html” } mock.once.expect(”/index.html”).response { return [200, “Hello”] }
  2. Actual test comes here …

The mock framework will make sure that each of these requests happen exactly once and in that order. It will fail the test if ”/” was requested twice, or if ”/index.html” was never requested, or if somehow ”/index.xml” was requested.

It’s the easiest way to express the expected behavior of the server and test against it.

You can do the same to test console IO - replace STDOUT/STDIN with something that looks like IO - SFTP, etc. The hardest part is setting up the first test, I spent more than a day on HTTP GET alone. But after that, the rest of the tests just flow

The ruby cookbook ch 17.9 has an example using flexmock. Also most of the packages have decent examples. I actually learned the skill from easymock when I was doing some heavy duty Java server programming.

I agree that mocking the low level code to test the high level code is the way to go, but something I’ve been struggling with lately is that at some point, I actually need to do remote calls to test the lower level code, and make sure the server responds like I think it does.

For example, I’ve been working on a CalDAV client recently, and while I’m trying to mock as much as possible, using the CalDAV spec as a guide, I really should be doing some remote tests against real servers to make sure they comply with the spec, and make sure I understand it.

(Assaf: check out FakeWeb for mocking HTTP. It does basically the same thing you described in the first half of your comment. It would be nice to see it get the ability to use mocks/expectations.)

Brandon, thanks for the tip. FakeWeb was exactly what I needed early on. It doesn’t do mocks, though. A MockFakeWeb would be a killer.