27 Oct 2006

Mining ActiveSupport: Object#returning

Posted by Jamis on Friday, October 27

ActiveSupport is the unsung hero of Rails. Much of what makes Rails code beautiful and powerful is implemented in ActiveSupport. It adds numerous methods to many of the core classes, which are used throughout Rails. If you don’t mind digging around, you can find all kinds of handy tricks to use in your own applications.

Consider this common idiom:

1
2
3
4
5
6
7
def create_book
  book = Book.new
  book.title = "Trafalgar: The Nelson Touch"
  book.author = "David Howarth"
  @db.add(book)
  book
end

You create an object, set some properties on it, and then return the object. ActiveSupport adds a new method (called returning) to Object, which can be used to beautify (and in some cases, simplify) that idiom:

1
2
3
4
5
6
7
def create_book
  returning Book.new do |book|
    book.title = "Trafalgar: The Nelson Touch"
    book.author = "David Howarth"
    @db.add(book)
  end
end

The latter is no shorter than the former, but it reads more elegantly. It feels more “Rubyish”. And there are times that it can save you a few lines of code, if that’s important. (Just scan the Rails source code for more examples.)

Object#returning has been in Rails for quite some time, being checked in on March 20, 2005 in revision 949. (It now lives in active_support/core_ext/object/misc.rb, and behaves slightly differently than the original did, but it is still as brief as ever.)

Looking at the comment for that method, you’ll see it described as “A Ruby-ized realization of the K combinator, courtesy of Mikael Brockman.”

In researching this article, I googled (pardon me, “searched Google”) for Mikael Brockman, to see just who this genius is. Unfortunately, though he seemed active enough in 2005, he appears to have fallen off the face of the Internet since then. At any rate: big props to Mr. Brockman for contributing this.

If you’re into functional programming at all, you probably know right away what the K combinator is. For the rest of us, this article is helpful (if a bit on the academic side, but that’s about par for the course for most reading about functional topics). Basically, the K combinator is a function of two arguments, that merely returns the first argument. The second argument is useful only for the side-effects it has on the first argument. The implementation of this in Ruby is beautifully succinct:

1
2
3
4
def returning(value)
  yield(value)
  value
end

It takes a value (the first argument), yields it to the associated block (the implicit second argument), and then returns the value. The block is used to operate on the first argument, and thus (by its side-effects) provides the actual work of this method.

It’s pretty handy!

Posted in Spotlight | 9 comments

20 Sep 2006

Introducing the Capistrano Shell

Posted by Jamis on Wednesday, September 20

Our current cluster at 37signals consists of (at the moment) 12 machines. The first few weeks that we were running them live were rather bumpy, and I took to using Capistrano to do a bit of ad-hoc monitoring, using the uptime task (from the capistrano-ext gem) to keep an eye on things:


cap -v uptime

However, with 12 machines, the wait for that command to run each time was about 20 to 30 seconds, which is far too long for my impatient self.

Part of that seemingly-interminable wait was alleviated by changing Capistrano so that it connects to servers in parallel, rather than serially. Prior to 1.2, when you executed a task that needed to connect to multiple servers, each connection was established one at a time, in a single thread. Now, they are established in parallel, one per thread. The connection overhead, which used to be 18 seconds on my laptop, dropped to just over 6 seconds. Not a bad savings!

However, there was more that could be done. Why, when I am running this command every few minutes, should I need to reestablish the connection with every request? What if I could cache the connection somehow?

This line of thinking led to what I feel is the most exciting new feature in Capistrano 1.2: cap shell.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
jamis> cap -v shell
====================================================================
Welcome to the interactive Capistrano shell! This is an experimental
feature, and is liable to change in future releases.
--------------------------------------------------------------------
cap> !uptime
[establishing connection(s) to db2, db1, db3, file1, app1, app2, app3, app4, app5, web1, web2, web3]
querying servers...
[app1 ] up   15 days, 7:40, load 3.53, 1.36, 0.83
[app2 ] up  15 days, 8 hrs, load 0.23, 0.38, 0.44
[app3 ] up  19 days, 11:13, load 0.29, 0.40, 0.43
[app4 ] up  19 days, 12:02, load 0.58, 0.49, 0.51
[app5 ] up   20 days, 9:50, load 0.50, 0.45, 0.43

[db2  ] up  28 days, 22:29, load 0.05, 0.06, 0.07

[file1] up  60 days, 12:42, load 0.06, 0.17, 0.17

[web1 ] up  19 days, 22:31, load 0.17, 0.12, 0.13
[web2 ] up  36 days, 22:18, load 0.06, 0.10, 0.11

[db1  ] up  28 days, 21:07, load 1.27, 0.90, 0.82
[db3  ] up  28 days, 22:53, load 1.19, 0.94, 0.85

[web3 ] up 188 days, 19:36, load 0.09, 0.06, 0.01
cap> 

Subsequent invocations of the uptime task from within that shell will reuse the existing connections. The result? It now takes less than two seconds to give me my answers.

But the shell is good for more than just executing tasks. You can execute arbitrary commands and have them run on your servers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cap> uname -srp
[db2] FreeBSD 6.1-STABLE amd64
[web2] FreeBSD 6.1-STABLE amd64
[web3] FreeBSD 6.0-RELEASE amd64
[file1] Linux 2.6.9-34.0.2.ELsmp x86_64
[app1] FreeBSD 6.0-RELEASE amd64
[app2] FreeBSD 6.1-STABLE amd64
[app3] FreeBSD 6.1-STABLE amd64
[app4] FreeBSD 6.1-STABLE amd64
[app5] FreeBSD 6.1-STABLE amd64
[web1] FreeBSD 6.1-STABLE amd64
[db1] FreeBSD 6.1-STABLE amd64
[db3] FreeBSD 6.1-STABLE amd64
cap> 

(Note that prefixing the command with an exclamation point causes Capistrano to interpret it as the name of a task, and to execute it as such. Without the exclamation mark, it is considered a bare command, and is simply executed verbatim on each host.)

Also, you can focus the task or command so that it only executes on a specific role, or host:

1
2
3
4
5
6
7
8
9
10
cap> with app uname -srp
[app1] FreeBSD 6.0-RELEASE amd64
[app2] FreeBSD 6.1-STABLE amd64
[app3] FreeBSD 6.1-STABLE amd64
[app4] FreeBSD 6.1-STABLE amd64
[app5] FreeBSD 6.1-STABLE amd64
cap> on web1,web2 uname -srp
[web1] FreeBSD 6.1-STABLE amd64
[web2] FreeBSD 6.1-STABLE amd64
cap> 

You can use with role to focus the command to all machines answering to the named role. Use a comma-delimited list to execute on machines in any of a list of roles.

If that is too general, you can get as specific as you wish using on host. This lets you execute only on the named host, or hosts. In fact, you can name hosts this way that aren’t even defined in your deploy.rb—it’ll establish connections on the fly to any machines it doesn’t recognize.

Lastly, you can use both with and on to set the scope for subsequent commands. Just leave the command off:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cap> with app
scoping with app
cap> uname -srp
[app1] FreeBSD 6.0-RELEASE amd64
[app2] FreeBSD 6.1-STABLE amd64
[app4] FreeBSD 6.1-STABLE amd64
[app3] FreeBSD 6.1-STABLE amd64
[app5] FreeBSD 6.1-STABLE amd64
cap> hostname -s
[app1] 82095-app1
[app4] 82098-app4
[app5] 82099-app5
[app2] 82096-app2
[app3] 82097-app3
cap> with all
scoping with all
cap> hostname -s
[db2] 82101-db2
[web3] 82094-web3
[app1] 82095-app1
[app2] 82096-app2
[db1] 82100-db1
[file1] 82103-file1
[app3] 82097-app3
[app4] 82098-app4
[app5] 82099-app5
[web1] 82092-web1
[web2] 82093-web2
[db3] 82102-db3
cap> 

(Note the use of the special all keyword, to return the scope to all roles and machines.)

So, armed with this new tool, I was able to keep an eye on the load of each machine in the cluster using the uptime task. When I noticed something anomalous, I focused on the box in question and looked for rogue processes:

1
2
3
4
5
6
cap> on app3 ps waux | head -n 5
...
cap> on app3 sudo kill 12345
Password:
...
cap>

It worked great. It’s definitely not a substitute for a real SSH shell, but it’s perfect for quick-and-dirty tasks that require you to switch between hosts frequently.

Which begs the question: why isn’t it a substitute for a real SSH shell?

Well, firstly, cap shell is stateless. Each command is executed in a new shell on the remote host. This means that commands like cd and export are pretty useless, since they won’t stick.

Secondly, cap shell isn’t intended to deal with interactive commands. You can’t, for instance, run IRB on multiple hosts simultaneously using cap shell. It does manage to deal with commands like tail -f, but that’s about the limit.

Thirdly, if you thought rm -rf / was dangerous when connected to a single host, imagine the damage you could do with cap shell! This is probably one of the biggest reasons it is still experimental. Until I can find a way to make it less likely to accidentally wipe an entire cluster with a single command, you ought to go into using this with caution.

Still, even with all those caveats, cap shell has become an irreplaceable tool in my toolbox. I’d love to hear from you with ideas for how to make it better, and safer.

Posted in Spotlight | 0 comments

27 Mar 2006

Web services, Rails-style

Posted by Jamis on Monday, March 27

The new REST web-service support in Rails 1.1 makes it so easy to add web-services to your app, you might as well do it earlier, rather than later.

Consider: if you have a page in your app that displays a list of people, it might look something like this, without web-service support:

1
2
3
def list
  @people = Person.find(:all)
end

Here’s the same action, with web-service support baked in:

1
2
3
4
5
6
7
8
def list
  @people = Person.find(:all)

  respond_to do |wants|
    wants.html
    wants.xml { render :xml => @people.to_xml }
  end
end

What that says is, “if the client wants HTML in response to this action, just respond as we would have before, but if the client wants XML, return them the list of people in XML format.” (Rails determines the desired response format from the HTTP Accept header submitted by the client.)

Now, let’s suppose you have an action that adds a new person, optionally creating their company (by name) if it does not already exist. Without web-services, it might look like this:

1
2
3
4
5
6
def add
  @company = Company.find_or_create_by_name(params[:company][:name])
  @person  = @company.people.create(params[:person])

  redirect_to(person_list_url)
end

Here’s the same action, with web-service support baked in:

1
2
3
4
5
6
7
8
9
10
11
def add
  company = params[:person].delete(:company)
  @company = Company.find_or_create_by_name(company[:name])
  @person  = @company.people.create(params[:person])

  respond_to do |wants|
    wants.html { redirect_to(person_list_url) }
    wants.js
    wants.xml  { render :xml => @person.to_xml(:include => @company) }
  end
end

It was simple enough that I also added RJS support here. If the client wants HTML, we just redirect them back to the person list. If they want Javascript (wants.js), then it is an RJS request and we render the RJS template associated with this action. Lastly, if the client wants XML, we render the created person as XML, but with a twist: we also include the person’s company in the rendered XML, so you get something like this:

1
2
3
4
5
6
7
8
9
<person>
  <id>...</id>
  ...
  <company>
    <id>...</id>
    <name>...</name>
    ...
  </company>
</person>

Note, however, the extra bit at the top of that action:

1
2
company = params[:person].delete(:company)
@company = Company.find_or_create_by_name(company[:name])

This is because the incoming XML document (if a web-service request is in process) can only contain a single root-node. So, we have to rearrange things so that the request looks like this (url-encoded):


person[name]=...&person[company][name]=...&...

And, like this (xml-encoded):

1
2
3
4
5
6
<person>
  <name>...</name>
  <company>
    <name>...</name>
  </company>
</person>

In other words, we make the request so that it operates on a single entity—a person. Then, in the action, we extract the company data from the request, find or create the company, and then create the new person with the remaining data.

Note that you can define your own XML parameter parser which would allow you to describe multiple entities in a single request (i.e., by wrapping them all in a single root note), but if you just go with the flow and accept Rails’ defaults, life will be much easier.

Posted in Spotlight | 21 comments

08 Mar 2006

Integration Testing in Rails 1.1

Posted by Jamis on Wednesday, March 8

Integration tests are a new feature of Rails 1.1 that take testing your applications to a new level. They are the next logical progression in the existing series of available tests:

  • Unit tests are very narrowly focused on testing a single model
  • Functional tests are very narrowly focused on testing a single constroller and the interactions between the models it employs
  • Integration tests are broad story-level tests that verify the interactions between the various actions supported by the application, across all controllers

This makes it easier to duplicate (in tests) bugs with session management and routing. Consider: what if you had a bug that was triggered by certain cruft accumulating in a user’s session? Hard to mimic that with functional tests.

Click here to read the rest of this article.

Posted in Spotlight | 30 comments

11 Oct 2005

Plugging into Rails

Posted by Jamis on Tuesday, October 11

So, you’ve got a new acts_as_chunky_bacon mixin you want to add to ActiveRecord, but those villianous core team members have turned you down and called you unworthy. What other options do you have? People need this mixin. You understand that in Ruby, things ought to act like chunky bacon.

It used to be (in that dim age before the 2005 RubyConf) that your only recourse was to package the thing up and tell people to go through the hassle of putting your mixin somewhere, adding it to their load-path, and requiring the file.

No more! Edge Rails now sports a very-simple-but-effective plugin system. You, as the author, can now give someone a zip file and tell them to simply uncompress it into their vendor/plugins directory—and that’s it. No more configuration required by the user.

All you, as the author, need to do is create a project with the following directory structure:

  acts_as_chunky_bacon/
  acts_as_chunky_bacon/init.rb
  acts_as_chunky_bacon/lib
  acts_as_chunky_bacon/lib/acts_as_chunky_bacon.rb

When the application starts, the lib directory will be automatically added to the load path, and the init.rb automatically loaded. (Either may be absent.) The init.rb just needs to do something like the following:

  ActiveRecord::Base.send :include, ActsAsChunkyBacon

What does this mean? It means that consumers of your plugin only need to drop your project in their vendor/plugins directory, and then start applying it to their own model objects:

  class PoignantGuide < ActiveRecord::Base
    acts_as_chunky_bacon :from => "chapter 3" 
    ...
  end

Currently, we at 37signals are using this plugin system to share code between our various applications. We are using it for things like email notification on errors, or common before_filter’s, or our web service infrastructure that allows Backpack and Basecamp to integrate with Writeboard.

I really like this new plugin system. It probably isn’t perfect, yet-I’m sure people will find ways to make it even handier-but it really makes it a lot easier to share code. Hopefully it will also make it easier for the Rails core team to say “no” to many proposed new features, since many of them can now be more easily shared as third-party additions.

Posted in Spotlight | 19 comments