Dynamic finders are one of my favorite bits of syntactic sugar in Rails. Suppose you have a form that people can use to log into your application. It submits the username and a password, and you then need to take those data and find the corresponding user.

Although there are several ways to do this, the best way is to use a dynamic finder, like so:

1
2
3
4
5
6

user = User.find_by_username_and_password(params[:username], params[:password])
if user.nil?
  # no such user, let them retry
else
  # log them in
end

In this next installment of “Under the Hood: ActiveRecord::Base.find”, we’ll explore the implementation of dynamic finders.

→ Click here to read the rest of this article.

When working with Rails, it is tempting to conceptualize ActiveRecord#find into some genie that magically reaches into your database and pulls back the records matching the criteria you specified. The reality is a bit more complex, and sadly lacking in genies.

To satisfy any request, #find works in two stages. First, it has to convert your criteria into SQL. Then, it has to take the results returned from the database and instantiate them as ActiveRecord objects.

The process of converting criteria into SQL is surprisingly complex, even for the simple cases. Let’s walk through the code to get an idea of what happens for the most trivial of cases, User.find(1).

(I’m working off of edge rails here, which at the time of this writing was at revision 5585. Code snippets given in this article will inevitably diverge from the latest edge. Caveat lector!)

→ Click here to read the rest of this article.

People attribute a lot of magic behavior to ActiveRecord::Base.find, and rightly so. It does a lot of different things. Want to find a record by id? Done. Multiple records by id? Done. Records based on custom conditions, including associations (arbitrarily deep), and limit the result? Done, done, and done.

However, all this magic comes at a price: the implementation of find is non-trivial.

I’d like to help people better understand how find works. By understanding what it does and how it does it, you’ll be able to make better decisions about your own applications, such as when eager loading of associations might be a bad idea, or what kinds of indexes your tables need.

So, next week (Thanksgiving notwithstanding) I’ll work on getting the first installment out. I learned one very important thing from the last “Under the hood” series I did: don’t write monstrously long articles, because most people won’t read them. So, this series will consist of much shorter, more bite-sized looks at specific parts of the #find implementation. Stay tuned!

Last Friday, I introduced my routing tricks plugin for Rails, by walking through the implementation of a routing extension that let you specify HTTP redirections via routes. Today’s article extends that plugin with a new feature: recognition by host, domain, or subdomain.

Suppose, for instance, that I wanted my blog’s admin feature to live at its own subdomain, admin.jamisbuck.org. Currently, that means I’d need the action that maps to ”/” to determine how to proceed based on the subdomain. Using this plugin, though, I could simply define my routes like this:

1
2
3
4
5
6

ActionController::Routing::Routes.draw do |map|
  map.connect "/", :controller => "admin",
    :conditions => { :subdomain => "admin" }
  map.connect "/", :controller => "blog"
  # ...
end

Note that I’ve defined two routes that map to ”/”, but the first is constrained by the subdomain. Any request for ”/” that comes in with a subdomain of “admin” will be routed to the admin controller. If the subdomain is not “admin”, the “blog” controller will be used instead.

You can do the same thing with :host and :domain, and any three of them may be regexes if you want that kind of flexibility.

Nifty! However, let’s get to the point of this article: how does this extension work its magic? Though not as trivial as the last article, it’s still remarkably simple.

→ Click here to read the rest of this article.

Assuming you’ve been reading this blog for the last couple of weeks, you’ve followed as we’ve explored Rails’ routing DSL implementation, peeked into the nooks and crannies of the route recognition code, and (most recently) went spelunking into the very bowels of route generation.

It’s time to begin putting all that reading to some practical, hands-on use.

We’ll start with something almost trivial: adding a “redirect” feature to routing, such that you can have any request to a particular route automatically respond with a 302 that sends the caller to another route.

Why would we want this? Well, besides the obvious pedagogical application, consider the situation of RESTful routes:

1
2
3

ActionController::Routing::Routes.draw do |map|
  map.resources :people
end

The map.resources command installs a whole slew of routes for you. However, it does not do anything with the root URI (the one that’s just a slash, ”/”). For most RESTful applications, you typically want that ”/” URI to map to one of your primary resources, like ”/people” in the above example. Normally, you’d just add another route like:

map.connect "/", :controller => "people", :action => "index"

That feels a little less-than-DRY. It would be nicer to just say something like:

map.redirect "/", :people

Here’s how we’ll do it.

→ Click here to read the rest of this article.

I previously wrote about the implementation of the routing DSL in Rails, as well as the internals of route recognition. If you haven’t yet read (and understood) both of those articles, I strongly encourage you to do so before delving into this one, since it builds on the material presented in those.

Route generation is the last stop on this tour of the Rails routing code. It also happens to be the hairiest, most mysterious, and most difficult bit, so make sure you’ve got a recent version of the rails code handy to follow along with (this article outlines the implementation as of revision 5304). Also, don’t be afraid to take this article a bit at a time, and to go over it repeatedly. It’s dense stuff, and unless you’re already familiar with the routing code, it might be hard to swallow in one (or even two) sittings.

→ Click here to read the rest of this article.

Monday’s article presented the implementation of Rails’ routing DSL. (If you haven’t read it yet, you ought to—this article assumes you’re familiar with at least as much of the routing code as that prior article explained.)

The DSL implementation of routes only scratches the surface. In this second installment, we’re going to delve even deeper. We’re going to lay bare the mysteries of route recognition.

Route recognition is one of the very first tasks that a Rails application executes upon receiving a request. What it does is (conceptually) very simple: given a URI path, determine what controller and action should process the request, as well as what additional parameters should be passed in. In practice, however, there’s a lot of complexity hidden there.

The journey begins in railties/lib/dispatcher.rb, in Dispatcher.dispatch. First, the request and response objects are created, the application is “prepared” (with actions that vary depending on whether you are running in production mode or not), and then routing is asked to recognize the current path.

1
2
3
4

request, response = ActionController::CgiRequest.new(cgi, session_options),
  ActionController::CgiResponse.new(cgi)
prepare_application
controller = ActionController::Routing::Routes.recognize(request)

With that innocent command, we leap into the routing code. Feel free to follow along, beginning on line 1243 of routing.rb:

1
2
3
4
5

def recognize(request)
  params = recognize_path(request.path, extract_request_environment(request))
  request.path_parameters = params.with_indifferent_access
  "#{params[:controller].camelize}Controller".constantize
end

That first line of the recognize method first extracts an “environment” hash from the request, and then invokes recognize_path with the path from the request, and the environment hash. This environment hash currently consists of only the request method, but if you are writing a routing extension that needs other information from the request (like the host name, or whether HTTPS is enabled) you can extend the RouteSet#extract_request_environment method to pull the additional data out. You’ll see (later) where that information is used in the recognition process.

The RouteSet#recognize_path method simply iterates over all defined routes, asking each whether or not it can recognize the given path. As soon as one responds in the affirmative, the loop stops and the result is returned. If no route matches the given parameters, a RoutingError is raised.

1
2
3
4
5
6
7

def recognize_path(path, environment={})
  path = CGI.unescape(path)
  routes.each do |route|
    result = route.recognize(path, environment) and return result
  end
  raise RoutingError, "no route found to match #{path.inspect} with #{environment.inspect}"
end

Here, then, is where things begin to get interesting. Go ahead and jump to Route#recognize, on line 464:

1
2
3
4

def recognize(path, environment={})
  write_recognition
  recognize path, environment
end

“But wait!” you say. “There’s nothing there but a recursive call to Route#recognize!”

“Ah,” I reply, “but note the call to write_recognition...”

Thus we introduce one of the reasons the routing code is hard to grasp. It rewrites itself on demand, for optimization reasons. Basically, the first time a route is asked to recognize a path, it will take all of its component segments, and all of their requirements, and dynamically generate a new recognize method based on them. Subsequent calls to that route’s recognize method will use the dynamically generated version. This allows route recognition to be quite fast, even with many routes defined.

That’s not much comfort, however, to the stalwart spelunker who wishes to understand how it all works.

Let’s try to demystify this by looking first at what a few dynamically generated recognize methods look like. From there, we can better understand the steps which the routing code takes to actually build that code.

Specifically, let’s consider the following three routes:

1
2
3
4
5
6
7
8
9
10

ActionController::Routing::Routes.draw do |map|
  map.connect "/", :controller => "foo", :action => "index"

  map.connect "/foo/:action", :controller => "foo"

  map.connect "/foo/:view/:permalink", :controller => "foo",
    :action => "show", :view => /plain|fancy/,
    :permalink => /[-a-z0-9]+/,
    :conditions => { :method => :get }
end

If you could see the code that gets generated for that first route, you’d see that it’s new recognize method would look more or less like this:

1
2
3
4
5
6

def recognize(path, env={})
  if (match = /\A\/?\Z/.match(path))
    params = parameter_shell.dup
    params
  end
end

In other words, match the path against the given regex (testing only to see if the string is empty, or a forward slash) and if it succeeds, return the route’s parameter_shell. (The parameter shell is the list of all non-regex requirements for a given route; in this case, it will be :controller => "foo", :action => "index", because those are the options that were given in the route’s definition.)

That’s the simplest case. Moving to the next route, we can see how dynamic segments like :action get handled:

1
2
3
4
5
6
7

def recognize(path, env={})
  if (match = /\A\/foo(?:\/?\Z|\/([^\/;.,?]+)\/?)\Z/.match(path))
    params = parameter_shell.dup
    params[:action] = match[1] || "index"
    params
  end
end

Again, the first thing that happens is the path is matched against a regex. The regex simply makes sure the path begins with ”/foo”, and is followed by an optional group that contains anything except path delimiters. (In this case, the group is optional, because the :action key is always defaulted to “index”. Other keys, as you’ll see, are not necessarily optional.)

If the regex matches, we dup the parameter shell, and then set the :action parameter to either the first match, or “index”. Then, the parameters are returned.

Pretty straightforward! Let’s move on to the third and final example, which looks like it might be a lot more complex. We’ve got two keys in the path (:view and :permalink), both of which have regex that restrict the set of values they can match. We also require that the route only match if the request method is GET. Behold:

1
2
3
4
5
6
7
8

def recognize(path, env={})
  if (match = /\A\/foo\/(plain|fancy)\/([-a-z0-9]+)\/?\Z/.match(path)) && conditions[:method] === env[:method]
    params = parameter_shell.dup
    params[:view] = match[1] if match[1]
    params[:permalink] = match[2] if match[2]
    params
  end
end

It just doesn’t get much simpler than that, folks. We match the path against the regex, and we compare the request method that the route requires (in the conditions hash) against the request method that was actually used (in the env hash). If all is good, we populate the params with the :view and :permalink values that were extracted from the path, and return it.

Boom! (As Steve Jobs would say.)

So, now we have some idea of the code that we want to generate. The rest of this article will show how it is actually built.

First, take a look at the Route@write_recognition method on line 370.

1
2
3
4
5
6
7
8
9

def write_recognition
  body = "params = parameter_shell.dup\n#{recognition_extraction * "\n"}\nparams"
  body = "if #{recognition_conditions.join(" && ")}\n#{body}\nend"

  method_decl = "def recognize(path, env={})\n#{body}\nend"

  instance_eval method_decl, "generated code (#{__FILE__}:#{__LINE__})"
  method_decl
end

All it does is build up a string that contains the method definition, and then sends it to instance_eval to actually install the new method. It also returns the string, so you can debug your routes easily by doing something like:

1
2
3
4
5

ActionController::Routing::Routes.routes.each do |route|
  puts route
  puts route.write_recognition
  puts
end

Go ahead and try that—it’s quite educational!

The write_recognition method builds the method in three parts:

1. the “body” (what gets executed when the regex matches) via recognition_extraction.
2. the “conditions” (the regex and any other special conditions) via recognition_conditions.
3. the “method declaration” (the method name and parameters)

Let’s look at how the body gets built first. Go ahead and jump to line 401, Route#recognition_extraction.

1
2
3
4
5
6
7
8
9

def recognition_extraction
  next_capture = 1
  extraction = segments.collect do |segment|
    x = segment.match_extraction next_capture
    next_capture += Regexp.new(segment.regexp_chunk).number_of_captures
    x
  end
  extraction.compact
end

What this does is loop over all the segments that compose the route. Each segment is asked for a string containing Ruby code that will extract the necessary information for that segment. These snippets of code are then collected into an array, and nil entries eliminated (via Array#compact).

I hate to do this to you, gentle reader, but let’s skip down one more level in the call stack and look at one of the match_extraction implementations. The default Segment#match_extraction method just returns nil—by default a segment encapsulates no parameter data. However, segments like DynamicSegment and ControllerSegment contain information that needs to be extracted. Let’s just look at DynamicSegment#match_extraction (on line 716):

1
2
3
4
5
6

def match_extraction(next_capture)
  hangon = (default ? "|| #{default.inspect}" : "if match[#{next_capture}]")
  
  # All non code-related keys (such as :id, :slug) have to be unescaped as other CGI params
  "params[:#{key}] = match[#{next_capture}] #{hangon}"
end

Here, “hangon” is just a cute variable name for a snippet of code that trails the match assignment (like a default value, or a conditional capture). Note also the next_capture parameter; this is used to keep track of the which capture (or captures) to extract from the match parameter.

Though I won’t go into them here, the match_extraction methods for both ControllerSegment and PathSegment are similar.

One last thing to point out in recognition_extraction: the call to Regexp#number_of_captures. This method is defined near the top of the routing.rb file, and it simply returns the number of capture groups within the regular expression. This is used to determine which capture indexes to allocate to each segment (in match_extraction), since a segment cannot pull data from capture groups it did not define.

Alright, following this so far? We’re almost done. Let’s next look at how the regex itself is constructed, and how conditions like the request method comparison are built.

1
2
3
4
5
6
7
8
9
10
11
12
13

def recognition_conditions
  result = ["(match = #{Regexp.new(recognition_pattern).inspect}.match(path))"]
  result << "conditions[:method] === env[:method]" if conditions[:method]
  result
end

def recognition_pattern(wrap = true)
  pattern = ''
  segments.reverse_each do |segment|
    pattern = segment.build_pattern pattern
  end
  wrap ? ("\\A" + pattern + "\\Z") : pattern
end

What this does is first construct a regular expression to compare against the path. This is done by aggregating the patterns of each segment into a single regular expression (via the recognition_pattern method), and then appending the request method comparison (if relevant for this route). For those of you wanting to extend routing with your own custom conditions (like routes based on hostname and such), this is where you would add those conditions, based on the conditions hash.

So! We’ve now generated the code for the route. Hiking back up the call stack, we find ourselves back in write_recognition, which evaluates the string and installs the new method into the route. Hiking up another level, we wind up back in the original Route#recognize method, where we make what appears to be a recursive call to Route#recognize. However, this will actually invoke the new method definition, recently installed by write_recognition, which will execute the newly generated code.

And there you have it, ladies and gents, the route recognition code. It’s really not so much of a much, is it? Once you wrap your mind around run-time generation of code, it all flows together pretty well. There are some edge cases and such that I didn’t cover, but you’re encouraged to explore those on your own. “An exercise for the reader,” and all that. Especially, try investigating what a route looks like that has optional values (:permalink => nil), or which uses path segments. See what the recognition code for such routes consists of.

By this point, you should have some grasp of about two-thirds of the routing code. The remaining third, route generation, will be covered in the next article, but be warned: it’ll be the hairiest of the three!

Rails is chock full of magic. From ActiveRecord to ActionView, the full stack employs just about every Ruby idiom in the book to make the programming experience as smooth, painless, and seamless as possible. This comes at a price, though: the source code is generally pretty opaque to Ruby novices. We’ve done our best to keep things readable (it’s in our best interest, after all, since the easier it is to read, the easier it is for people to create and submit patches), but there are still certain areas of Rails that are widely regarded as “hard to follow”.

Routing is one of those areas.

This is the first of three articles that will delve into the dark recesses of the routing code. It deals only with the implementation of the routing DSL (e.g., the part you use in config/routes.rb). The next two articles will deal with route recognition, and route generation, respectively.

I’d encourage you to follow along in the routing code. This article covers the version of routing in edge rails. You’ll find that it bears very little relation to the version of routing in Rails 1.1.6 and earlier, but don’t let that throw you. As of this writing, revision 5169 contains the latest version of routing.rb.

The config/routes.rb file forms the primary (and generally, only) interface to routing for most Rails developers. If you look in the config/routes.rb file for the vast majority of Rails applications, you’ll see something like this:

1
2
3
4

ActionController::Routing::Routes.draw do |map|
  # ...
  map.connect ":controller/:action/:id"
end

That ActionController::Routing::Routes reference is rather misleading. It is not a class, but is simply a constant referring to an instance of ActionController::Routing::RouteSet. (You can see the instantiation occurring at the bottom of action_controller/routing.rb.) That one instance is in charge of managing the routes for the entire lifetime of the Ruby process.

1
2
3
4
5
6

module ActionController
  module Routing
    # ...
    Routes = RouteSet.new
  end
end

The RouteSet#draw method is where the DSL magic all begins. If you look in the routing.rb code (around line 1113), it’s only three lines long, but what a significant three lines those are!

1
2
3
4
5

def draw
  clear!
  yield Mapper.new(self)
  named_routes.install
end

That one method is the entry point for the entire routing DSL. First, any existing routes are removed from the collection (via the clear! method). Then, a new ActionController::Routing::RouteSet::Mapper instance is created and yielded to the block (where it is generally bound to the map variable in config/routes.rb). After all the routes have been “drawn” (or defined), any named routes are installed into ActionController::Base, so that the named routing helpers can be used by controller and view code (as foo_url and foo_path).

ActionController::Routing::RouteSet::Mapper is nearly trivial. It’s just a proxy that delegates to the RouteSet instance that created it. When you call map.connect, the work is delegated to RouteSet#add_route. When you create a named route, the method_missing hook on the Mapper redirects the call to RouteSet#add_named_route. If you are looking for ways to extend routing, take note: Mapper is what you need to extend in order to add to or change the routing DSL. For an example of how to extend it, take a look at action_controller/resources.rb. (That’s where the new RESTful routing options are defined.)

So, the call chain so far goes something like RouteSet#draw, Mapper#connect, RouteSet#add_route. Looking at RouteSet#add_route (line 1147), you’ll see it’s another tiny method of only three lines. Instead of doing all the work itself, it just calls on builder.build to create the new Route instance, and then adds the new route to the routes collection.

1
2
3
4
5

def add_route(path, options = {})
  route = builder.build(path, options)
  routes << route
  route
end

Looking at the definition for the builder method (line 1109), it just lazily instantiates an ActionController::Routing::RouteBuilder object and returns it. RouteBuilder is a factory class for creating new Route instances. You’ll find it defined beginning at line 785.

1
2
3

def builder
  @builder ||= RouteBuilder.new
end

The RouteBuilder is another good class to note if you are trying to extend Rails. The fact that RouteSet uses a builder method to lazily instantiate the builder means you can easily subclass RouteBuilder and then install your subclass using an overridden RouteSet#builder method. (You might want do this if, for instance, your routing extension adds a new kind of routing segment that needs special consideration during parsing.)

The main job of RouteBuilder is to take a path string, and a hash of options, and return a Route that corresponds to them. It does this by calling segments_for_route_path (line 807) to decompose (or tokenize) the path into “segments”. (Segments are the atomic substrings of the path, which represent the delimiters, static text, and dynamic tokens it contains.) The builder then calls assign_route_options to combine the default values and condtions with those segments. Sounds complex, but it’s actually remarkably straightforward.

You can see that segments_for_route_path just calls segment_for repeatedly to decompose the string. Each call to segment_for returns a new Segment instance that represents some section of the string.

Looking at segment_for, you can see that it just uses a case statement with regexen to determine what type of segment to return:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def segment_for(string)
  segment = case string
    when /\A:(\w+)/
      key = $1.to_sym
      case key
        when :controller then ControllerSegment.new(key)
        else DynamicSegment.new key
      end
    when /\A\*(\w+)/ then PathSegment.new($1.to_sym, :optional => true)
    when /\A\?(.*?)\?/
      returning segment = StaticSegment.new($1) do
        segment.is_optional = true
      end
    when /\A(#{separator_pattern(:inverted)}+)/ then StaticSegment.new($1)
    when Regexp.new(separator_pattern) then
      returning segment = DividerSegment.new($&) do
        segment.is_optional = (optional_separators.include? $&)
      end
  end
  [segment, $~.post_match]
end

(That returning statement may look odd; it’s a method defined in ActiveSupport that makes it really easy to return a value, but only after performing some operations on it. You’ll find it used all over in Rails, so it’s worth getting familiar with it.)

If you are trying to extend Routes, this is where your RouteBuilder subclass would extend segment_for to add it’s own custom string processing. As you can see, routing currently supports five different segment types:

• DynamicSegment. This represents parts of the route that begin with a colon, like :action, :permalink or :id.
• ControllerSegment. This is actually a subclass of DynamicSegment. It represents to special string :controller, because it does some special recognition on those strings. (We’ll cover that more in the next article).
• PathSegment. This is for segments that start with an asterisk, and which represent the remainder of the path. Routes like "/file/*path" use a PathSegment.
• StaticSegment. This is any static text in your route that must be matched (or generated) verbatim. If you have a path like "/one/two", the strings "one" and "two" are both static segments.
• DividerSegment. This is any segment that is used to delimit the other segments. Generally, this will be the forward slash character, but also includes commas, periods, semicolons, and question marks.

The new Route, once instantiated, will include an array of segments that encode the path it encapsulates. You can inspect the route by calling it’s #to_s method. That will give you a readable version of its path, and its options, should you ever need it. In fact, when I’m troubleshooting something, I find it helpful to add something like the following to the end of config/routes.rb:

1
2
3

ActionController::Routing::Routes.routes.each do |route|
  puts route
end

Rick Olson has also created a routing navigator plugin for Rails that makes it easy to see what routes exist in your project. This is especially handy if you are using the RESTful routes, since they dynamically generate a whole host of routes behind the scenes.

Before we conclude this whirlwind tour, we need to make one last brief stop. When you create a named route, the mapper delegates to the RouteSet#add_named_route method, which (after calling add_route) delegates to a helper object called named_routes. Each RouteSet instance includes a reference to a NamedRouteCollection instance (see line 986), called named_routes. Any time you add a route to this collection, a set of helper methods are automatically generated for that route and are added to an anonymous module, which is used to install the helpers into (e.g.) the ActionController::Base class. This installation only occurs when the NamedRouteCollection#install method is called, and that happens at the end of RouteSet#draw.

If you have any questions about something I glossed over (or omitted), please feel free to ask in the comments and I’ll try to answer.

That, then, is the overview of the implementation of the Routing DSL. The next article will deal with how this all ties into route recognition, which comes into play every time a request is received by a Rails application. Stay tuned!

Capistrano really is the poster child for Net::SSH. In the last “Inside Capistrano” article (the Gateway implementation) I talked about Capistrano’s use of Net::SSH’s port forwarding feature. This time around, I’d like to focus on how Capistrano uses Net::SSH to execute a single command in parallel on multiple hosts.

For now, we’re just going to skip past all the magic in Capistrano::Actor that manages the connections to the servers. (We’ll discuss that another time.) We’ll jump straight to Capistrano::Command, located in capistrano/command.rb. It acceps five arguments: a list of named servers, a command to be executed, a Proc instance to act as a callback for any output from the servers, a hash of options, and a reference to the Actor instance that requested the command. (Whew!)

The initialization is pretty straightforward:

1
2
3
4
5
6
7
8

def initialize(servers, command, callback, options, actor)
  @servers = servers
  @command = command.strip.gsub(/\r?\n/, "\\\n")
  @callback = callback
  @options = options
  @actor = actor
  @channels = open_channels
end

The most significant part of the initialization is the call to open_channels. For those of you unfamiliar with Net::SSH, every interaction with a remote host is encapsulated in a channel. Each connection can have multiple channels open simultaneously; it is this feature that lets you have multiple forwarded ports going over the same connection you are using to interact with your shell on the remote host. (Try doing that with telnet!)

Thus, in order to execute a command on the remote hosts, we need to open a channel for the command on each host. The open_channels method does just this. It’s not a complicated method, but if you aren’t familiar with Net::SSH, it might appear a little daunting with all the callbacks involved. We’ll break it up and take it a piece at a time.

First, we just iterate over each server, using map to return an array of channel objects that correspond to the servers. (We use the actor instance here to get at the actual Net::SSH sessions for each named server, so we can open those channels. It is assumed that each connection has been established previously.)

1
2
3
4
5
6
7

def open_channels
  @servers.map do |server|
    @actor.sessions[server].open_channel do |channel|
      ...
    end
  end
end

For each new channel, we do a bit of set up:

1
2
3

channel[:host] = server
channel[:actor] = @actor
channel.request_pty :want_reply => true

Every channel instance can be treated as a hash, so you can store custom data in it for later access. Here, we’re storing the name of the server the channel is connected to, as well as the actor reference (so we can use it in the callback). Then, we tell the remote host that we want to allocate a pty for this connection.

With that out of the way, we set up some callbacks to handle different channel events. These are detailed below, with a bit of commentary:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# The on_success handler is called when the server
# responds to our request_pty message, but only if
# a pty was allocated. We use this opportunity to send
# the actual command to the server, along with any data
# that should be piped to it on stdin.
channel.on_success do |ch|
  ch.exec command
  ch.send_data options[:data] if options[:data]
end

# Just as on_success is called when the server was able
# to allocate a pty, on_failure is called when it can't.
# In that case, we log a message and move on.
channel.on_failure do |ch|
  logger.important "could not open channel", ch[:host]
  ch.close
end

# Any time the remote command emits data on its stdout,
# Net::SSH will call the channel's on_data callback. We
# delegate to the callback hook given when the Command
# was instantiated.
channel.on_data do |ch, data|
  @callback[ch, :out, data] if @callback
end

# Stderr (and any other, non-stdout data) gets sent to
# the on_extended_data hook. We treat it all as stderr
# and delegate it to the primary callback.
channel.on_extended_data do |ch, type, data|
  @callback[ch, :err, data] if @callback
end

# The on_request hook is used for most other kinds of
# response from the server. All we care about is the
# 'exit-status' reply, which we use to determine whether
# or not the command completed successfully.
channel.on_request do |ch, request, reply, data|
  ch[:status] = data.read_long if request == "exit-status"
end

# When the command finishes, the on_close hook is called.
# We set a flag here that let's us easily query whether
# the channel is still active or not.
channel.on_close do |ch|
  ch[:closed] = true
end

Alright! The channels are all ready for us now, and we can proceed with executing the command. This occurs in the process! method, which has a bit of Net::SSH magic in it so that each channel is processed in parallel.

Each Net::SSH connection is event-driven, and as such requires an event loop to be running. Net::SSH gives you a method for running an event loop on a single connection (called “loop”), but if we want to drive multiple connections simultaneously, we need to implement our own event loop. That’s what the process! method does.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

def process!
  # First, we mark the current time. This is so that we
  # can ping each connection every second, so that
  # long-running commands don't result in the connection
  # timing out.
  since = Time.now

  # This begins the event loop...
  loop do

    # This indicates how many channels are still active.
    # When there are no more active channels, we can
    # terminate the event loop.
    active = 0

    # For every active channel, have it's associated
    # connection process any pending events. (The 'true'
    # parameter tells the poll not to block, if no
    # events are pending.)
    @channels.each do |ch|
      next if ch[:closed]
      active += 1
      ch.connection.process(true)
    end

    # If there aren't any active channels, break out of
    # the loop
    break if active == 0

    # If it has been at least a second since the last
    # ping, ping every connection. Note that we ping
    # whether the channel is active or not, since the
    # connection itself IS, and we don't want it timing
    # out just because one of the other channels is
    # lagging behind the others.
    if Time.now - since >= 1
      since = Time.now
      @channels.each { |ch| ch.connection.ping! }
    end
    
    # a brief respite, to keep the CPU from going crazy
    sleep 0.01
  end

  # If any command terminated with a non-zero exit
  # status, then we raise an exception. Ultimately,
  # Capistrano::Actor will catch that exception and try
  # to rollback the current task (if a rollback handler
  # is defined for it.)
  if failed = @channels.detect { |ch| ch[:status] != 0 }
    raise "command #{@command.inspect} failed on #{failed[:host]}"
  end

  self
end

When the command terminates, control reverts to the caller (the Capistrano::Actor instance). As you can see, there really isn’t that much to it—it just requires that we do a bit of manual labor to set up that custom event loop.

As with the Gateway code, you could probably mock up an actor instance and use the Command code independently of Capistrano, but it wasn’t really designed with that in mind. Still, it should provide plenty of inspiration for your own Net::SSH scripts.

If you’d like to learn more about Net::SSH, the manual is a good place to start.

Most Capistrano users have probably never needed to use its gateway feature. I find that vaguely ironic, since it was one of the features in Capistrano that were on the original list of requirements when I sat down to code it all up.

Basically, what the gateway lets you do is tunnel your connections through a single computer. This lets you connect to computers that are behind a firewall, or on a VPN. We use this feature all the time at 37signals, since the bulk of our cluster is not directly accessible via the Internet.

1
2
3
4
5
6
7
8

# specify the gateway server
set :gateway, "gateway.server.com"

# the following servers are behind a firewall and
# cannot be accessed directly
role :app, "app.server.com"
role :web, "web.server.com"
role :db,  "db.server.com", :primary => true

The gateway code is a bare 100 lines long, including comments. Basically, all it does is establish a connection to the gateway machine, and then for every connection established via the gateway, it forwards a port from the local host to the requested server. Then, it establishes a connection to the requested server via that forwarded port. It makes heavy use of threads to accomplish this, and is one of the places that helped iron out several synchronicity issues in Net::SSH. In fact, the code is a good showcase of what you can do with forwarded ports in Net::SSH.

So, let’s take it all apart and walk through the code, beginning with the initialize method. (For those of you that want to follow along, the file in question is capistrano/gateway.rb.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def initialize(server, config)
  @config = config
  @next_port = MAX_PORT
  @terminate_thread = false
  @port_guard = Mutex.new

  mutex = Mutex.new
  waiter = ConditionVariable.new

  @thread = Thread.new do
    SSH.connect(server, @config) do |@session|
      mutex.synchronize { waiter.signal }
      @session.loop { !@terminate_thread }
    end
  end

  mutex.synchronize { waiter.wait(mutex) }
end

(In the interest of keeping things compact, I’ve removed the comments and the lines related to logging.)

The meat of this is the Thread.new statement there in the middle. All it does is establish the gateway’s SSH connection. (The config instance variable is a Capistrano::Configuration instance, from which various SSH options are pulled, including the user, password, port, etc.) Once the connection is live, the block will be called, and we signal the “waiter” (the condition variable). This wakes up the calling thread (which is blocked in the wait call following the thread). Once the connection is live, we enter the session loop, which goes until asked to terminate (the terminate_thread instance variable).

Note that SSH.connect is another Capistrano abstraction that basically wraps the lower-level Net::SSH.start. There’s not much to it; you can read the entire thing in capistrano/ssh.rb.

Once the gateway connection is live, other connections may be established through it by calling the connect_to method, passing in a string that names the target server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

def connect_to(server)
  connection = nil
  port = next_port

  thread = Thread.new do
    begin
      @session.forward.local(port, server, 22)
      connection = SSH.connect('127.0.0.1', @config, port)
    rescue Errno::EADDRINUSE
      port = next_port
      retry
    rescue Exception => e
      puts e.class.name
      puts e.backtrace.join("\n")
    end
  end

  thread.join
  connection or raise "Could not establish connection to #{server}"
end

def next_port
  @port_guard.synchronize do
    port = @next_port
    @next_port -= 1
    @next_port = MAX_PORT if @next_port < MIN_PORT
    port
  end
end

For this bit, we first get the next (possibly) available port on the local host. Then, in a thread, we start a forwarded port from the local host to the remote host, and try to establish an SSH connection through it. If the port turns out to be in use, we grab the next port and try again.

And that’s it, really.

There isn’t that much to the gateway implementation, but we like it that way. It is one of the most critical parts of Capistrano for us at 37signals, and the current implementation is both simpler than before (compare it to the version in Capistrano 1.1) and more robust. You could even conceivably use the gateway code directly in your own scripts, if you ever needed to connect to one or more hosts through a forwarded port. Something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

require 'capistrano/gateway'
require 'capistrano/logger'

# First, we create a config object that quacks,
# mostly, like a Capistrano::Configuration object.
config = Struct.new(:user, :password, :ssh_options,
  :logger).new
config.user = "username"
config.password = "password"
config.ssh_options = {}
config.logger = Capistrano::Logger.new(
  :output => "/dev/null")

# Connect to the Gateway...
gateway = Capistrano::Gateway.new("gateway", config)

# Establish a connection to an internal machine via 
# the gateway
host = gateway.connect_to("internal")

# "host" is now an SSH session object. We can
# manipulate it using the Net::SSH API.
host.open_channel do |ch|
  ch.on_data do |ch, data|
    puts(data)
  end

  ch.exec "hostname"
end

host.loop

For more information on Net::SSH, you can tackle the Net::SSH documentation. It tries to be fairly comprehensive.

The Great Net::SSH Refactoring is nearing completion! I still need to add some unit tests for the KeyFactory class, but after that, it will be ready for a 0.5.0 release. One caveat—I’ll be releasing it without SFTP support.

Before anyone keels over from suspense, let me clarify that last remark. I am not abandoning SFTP support. Instead, I’m refactoring it out of the core Net::SSH deliverable, into it’s own library. It will (of course) depend on Net::SSH, but Net::SSH itself will no longer ship with SFTP support.

This isn’t as bad as it may seem, especially if you are using a package manager. If you are using a package manager, you’ll just go to install Net::SFTP, and voila, it will install Net::SSH as well (because that is a dependency).

This makes my life easier, since Net::SFTP was rapidly growing in complexity. It makes it easier to manage if it is its own library. It also means I can release the new version of Net::SSH sooner, since Net::SFTP is going to take about a week (or two) longer to refactor and repackage.

Besides that drastic side-effect of refactoring, I’ve also revisited how I did the transport, user-auth and connection layers. In particular, the transport session originally had to be explicitly opened after it was created. This was a side effect of when I had envisioned a single registry containing multiple Net::SSH connections—I’ve since abandoned that approach, and have required that a single registry contain no more than a single Net::SSH connection. (Well, if you’re clever you could still do multiple connections by managing namespaces right, but that would be more trouble than it’s worth.)

So, now that the transport session is a singleton, it made no sense to require the #open call to be made explicitly. Instead, now the various connection parameters and options are registered as services, and the session is opened automatically when the service is created.

This means that dependency injection can really shine, now. Consider the following scenario:

Net::SSH::Session is asked to open a new connection. It instantiates a registry and registers all the necessary services. Then, it grabs the user-auth service, to authenticate the user. This has a dependency on the transport session, so Needle instantiates the transport session first (which makes the connection, negotiates the algorithms to use, and so forth), and then instantiates the user-auth service. It then invokes the #authenticate method, gets the active connection service and returns.

I could have removed the explicit #authenticate call, as well, but I wanted to have more control over how a failed authentication is reported to the user. I suppose I could have registered some proc as a service, which when the authentication fails the proc gets called…but that’s complicating things unnecessarily.

I like the system much better now.

I’ll be releasing Net::SSH 0.5.0 next week, with any luck. Watch for it!

In spite of appearances, progress has been happening (slowly) on Net::SSH. This next installment of the “Refactoring Net::SSH” series will give a brief look at what the latest changes have been, and what’s next on my “to do” list.

Tweaking the Transport Layer

As you may recall, the last article in this series was about the completion of the refactoring of the transport layer. Complete or not, I’ve continued tinkering on it. Originally, I had intended to allow a single Needle registry to contain multiple SSH sessions.

I’ve changed my mind.

The reasons for this are several, but the primary one is that it is just much easier if you only allow singletons in your registry. (Or, rather, it is easier if you require the primary dependencies in your app to be singletons. I still have many prototype services in the new Net::SSH.) If a service is known to be a singleton, you can just inject it as a dependency into any other service you want, and you’re guaranteed that the same instance of that service will be used throughout your application. By making the transport session service a prototype service, instead of singleton, I made alot more work for myself, because I now had to manually inject the correct instance of the session into each service that required it.

So, the transport session is now a singleton, and a registry may only contain a single instance of it. This isn’t a big deal—it just means that when I get around to finishing Net::SSH, the primary client interface will need to create a new registry (behind the scenes, of course) for each SSH connection that is requested.

It turned out to be surprisingly easy to make the change, although there are still a few places left to tweak for consistency’s sake. All I had to do was change the definition of the transport session service, and then change how other services that were dependent on the session referenced that service. Took about 15 minutes.

User Authentication

Last week I finished refactoring the user authentication service. This included the ssh-agent interface, the user key manager, and the three supported authentication mechanisms (password, public-key, and host-based). Before, all of this was implemented in three classes: Agent, UserKeyManager, and UserAuth. I split it up even further, making (among other things) each supported authentication mechanism its own class. It is now all unit tested, and although I haven’t done a formal integration test with the transport layer, I did write a simple test script that seemed to work as expected.

(Which brings me to a question, for any integration-testing gurus that happen to read this. What would be the best way to test the integration of the user authentication stuff with the transport layer? The only way I can think of is to add a new user to my machine and hard code that user’s password in the integration tests (to test the password authentication mechanism). I would also need to enable host-based authentication in my ssh server, and enable this test user to log into my machine based on the host they are on. I hesitate to do either of these things, for security reasons…)

Next…

The next piece to tackle is the connection layer, which manages the opening, closing, and processing of SSH channels. This will be a monstrous bit of work, made more so because I don’t particularly like how it is currently implemented. Plus, I’ve forgotten the details of how it works, so I’ll need to hit the SSH spec again to refresh my memory of the particulars. Because I’m not too excited at the prospect of this, things have ground almost to a halt on the Net::SSH refactoring, although I did spend some time yesterday reacquainting myself with the current implementation.

I’m afraid I won’t be able to reuse very much of the existing implementation…and it’s close to 1,000 lines of code. Ouch. :(

I really want to finish this. There are several other projects I’d rather be working on!

After nearly a month of refactoring Net::SSH to take advantage of dependency injection (via the Needle framework) visible progress has been made: the transport layer works!

(A quick review: the transport layer of the SSH2 protocol deals with the low-level workings of SSH. It manages the algorithm negotiation, key exchange, encryption/compression of packets, and so forth. All other parts of the SSH protocol are built on top of the abstraction provided by the transport layer.)

How the Session is Initialized

As some of you may recall, the last article in this series ended on a question as I debated what approach to take regarding the initialization of the Session class. After all of that, Eivind Eklund suggested that I just pass the container to the Session constructor and let the Session initialize itself a la carte:

  container.register( :session, :model => :prototype ) do |c,p|
    Session.new( c.logs.get(p.fullname), c )
  end

At first, I was opposed to this approach, since it didn’t feel like dependency injection to me. The fact is, it isn’t. It’s the service locator pattern, which is closely related.

However, after some thought I realized that the price of being a purist in this case would mean more work for me. A good programmer should be able to use all of the tools at his disposal, understanding where they make sense and where they don’t. In this case, the service locator was a more efficient approach due to the large number of dependencies that Session had.

Further Refactoring, and Lessons Learned

The Session class itself was a beast (see the original sources). In order to tame it a bit and make it unit-testable, I moved the version and algorithm negotiation pieces into two different classes. This trimmed it down nicely, although it is still pretty big.

This most recent refactoring has finally hammered home a lesson that I feel ready to pen (though I’m certain others have already discovered and codified it elsewhere):

If you have a private method of a class that you feel needs unit testing, that method probably belongs in a class of its own.

In other words, suppose you have a private method foo that is called by a public method bar:

  class Agnathostomata
    def bar
      ...
      foo
      ...
    end

    def foo
      ...
    end
    private :foo
  end

When it comes time to test the class, you discover that you would like to unit test the foo method directly. Your options in this case are to (a) make foo public, or (b) circumvent the access control via send or instance_eval. Neither are very attractive options.

What I discovered is that it is probably better, in many cases, to move foo (and any related functionality) into a class of its own, making foo public in the process. Then, the first class delegates that functionality to the new class, like this:

  class Spiloma
    def foo
      ...
    end
  end

  class Agnathostomata
    def initialize
      @delegate = Spiloma.new
    end

    def bar
      ...
      @delegate.foo
      ...
    end
  end

This is exactly how the new VersionNegotiator and AlgorithmNegotiator classes came to be.

Results

I had been unit testing every piece this entire time, so I felt confident that there wouldn’t be any significant bugs. However, the pieces I hadn’t tested so far were the parts that dealt with the dependency injection. Naturally, when it came time to run the integration test, those were the parts that failed first. :)

The problems weren’t significant, however. Usually only a few typos or scoping issues. There was one place where I had indicated the maximum number of bits in the key should be 8196, when it should have been 8192, and that caused me a bit of grief, but I found and fixed it.

The integration test was simple: for every combination of cryptography backend, host key, kex algorithm, cipher algorithm, HMAC algorithm, and compression algorithm, run a test that opened a new connection via the transport Session, sent a message and received a response, and then closed the connection.

Voila!

  backends.each do |backend|
    keys.each do |key|
      kexs.each do |kex|
        encryptions.each do |encryption|
          hmacs.each do |hmac|
            compressions.each do |compression|
              method_name = "test_#{backend}__#{key}__#{kex}__#{encryption}__#{hmac}__#{compression}" 
              method_name.gsub!( /-/, "_" )

              define_method( method_name ) do
                @registry.register( :crypto_backend ) { backend }
                session = @registry.transport.session

                assert_nothing_raised do
                  session.open "localhost",
                    :host_key => key,
                    :kex => kex,
                    :encryption => encryption,
                    :hmac => hmac,
                    :compression => compression
                end

                assert_equal key, session.algorithms.host_key
                assert_equal kex, session.algorithms.kex
                assert_equal encryption, session.algorithms.encryption_c2s
                assert_equal encryption, session.algorithms.encryption_s2c
                assert_equal hmac, session.algorithms.mac_c2s
                assert_equal hmac, session.algorithms.mac_s2c
                assert_equal compression, session.algorithms.compression_c2s
                assert_equal compression, session.algorithms.compression_s2c

                type = nil
                assert_nothing_raised do
                  session.send_message "#{session.class::SERVICE_REQUEST.chr}\0\0\0\14ssh-userauth" 
                  type, buffer = session.wait_for_message
                end

                assert_equal session.class::SERVICE_ACCEPT, type 
                session.close
              end

            end
          end
        end
      end
    end
  end

The result?

  $ ruby test_integration.rb
  Loaded suite test_integration
  Started
  .........................................................................
  .........................................................................
  ..............
  Finished in 21.566316 seconds.

  160 tests, 1760 assertions, 0 failures, 0 errors

It’s alive!

Future Directions

Well, as I said earlier, the transport layer is just one piece (albeit a very fundamental one) of the SSH protocol stack. So, the parts that remain to be done are:

• User authentication
• Connection management (channels, etc.)
• SFTP protocol
• Various convenience interfaces

I figure I might be about half done with the entire refactoring. Regardless, once I’m done I’ll release Net::SSH 0.2.0, and will finally be able to proceed with all the parts that are still missing from it.

The adventure continues, as I persist in refactoring Net::SSH to take advantage of dependency injection, using Needle. This issue of the adventure will detail a snag in the refactoring, and some of my current thinking to overcome it.

I’m down to the last piece of the transport layer implementation. (For those that care, the transport layer is the lowest level of the SSH2 protocol, managing the sending and receiving of packets. This includes compression, encryption, and validation of packets.) This last piece is the Net::SSH::Transport::Session class.

The session is the piece that ties all the different parts of the transport layer together, sitting them all down at the table and making them work together. Specifically, the transport layer’s responsibilities include:

1. Establish a network connection to an SSH server.
2. Exchange version information with the other end of the connection.
3. Negotiate (with the remote node) the preferred algorithms to use for ciphers, key exchanging, compression and so forth.
4. Exchange keys
5. Provide an abstraction for sending and receiving packets

To accomplish these responsibilities, the session has dependencies on the following services:

• outgoing_packet_stream (for sending packets)
• incoming_packet_stream (for receiving packets)
• kex_names (for identifying the kex algorithm implementation to use)
• cipher_factories (for returning cipher implementations appropriate for the selected cryptography backend)
• key_factories (for returning key implementations)
• hmac_factories (for returning hmac implementations)
• buffer_factories (for returning buffer implementations)
• compression_algorithms (for returning compression algorithm implementations)
• decompression_algorithms (for returning decompression algorithm implementations)
• logs (for obtaining a new logger reference)
• socket_factory (for opening a network connection to the requested host/port)

There may even be a few more, as I dig deeper into refactoring this beast.

The challenge is this: nearly every one of those services is needed in the session’s constructor. That means that injecting the dependencies as setters won’t work—they don’t get set soon enough. Every single one of these must be passed as an argument to the constructor.

Yech.

So, I’ve spent some time thinking about this. There are a few different ways to go about this.

Cave. Just pass them to the constructor.

The first option is to just do it the “academic” way. “The process recommends approach X, so I’ll do approach X.”

I don’t think anyone will blame me from balking at this. How many parameters does that mean the constructor takes? Eleven?!

  transport.register( :session ) do |c,p|
    Net::SSH::Transport::Session.new(
      c.outgoing_packet_stream, c.incoming_packet_stream,
      c.kex_names, c.cipher_factories[c.crypto_backend],
      c.key_factories[c.crypto_backend], c.hmac_factories[c.crypto_backend],
      c.buffer_factories[c.crypto_backend], c.compression_algorithms,
      c.decompression_algorithms, c.logs.get(p.fullname), c.socket_factory
    )
  end

No.

True, the burden of this could be mitigated somewhat by sending a hash to the constructor and injecting the parameters that way. That almost makes it look like setter injection, too:

  transport.register( :session ) do |c,p|
    Net::SSH::Transport::Session.new(
      :outgoing_packet_stream => c.outgoing_packet_stream,
      :incoming_packet_stream => c.incoming_packet_stream,
      :kex_names              => c.kex_names,
      :cipher_factory         => c.cipher_factories[ c.crypto_backend ],
      ...
    )
  end

A little better. I might at least consider such an approach. But there’s got to be a better way!

Use setters and a custom ‘init’ method.

Constructor injection is fine-don’t get me wrong-but my personal preference is setter injection. When using constructor injection, it becomes hard to deal with many dependencies (as demonstrated above). Also, you can’t name the dependencies (unless you use a hash, which is too prone to typos).

So, another option I considered is to just use setter injection. The problem with this in this case, as I mentioned, is that the logic that needs the dependencies is in the constructor, which is called before the setters can be called.

The solution, then, is to put the construction logic that needs the dependencies in a separate method, which is invoked explicitly after the setters have been taken care of.

  transport.register( :session ) do |c,p|
    session = Net::SSH::Transport::Session.new
    session.outgoing_packet_stream = c.outgoing_packet_stream
    session.incoming_packet_stream = c.incoming_packet_stream
    session.kex_names              = c.kex_names
    session.cipher_factory         = c.cipher_factories[ c.crypto_backend ]
    ...
    session.initialize_service
  end

I like this better, because I can use my beloved setter injection. However, having to invoke the “initialize_service” method explicitly feels a bit kludgy. It is also prone to error, since that step would be easy to forget. (Fortunately, it would only need to exist in a very few places in the code, so that’s not as big of a drawback as it could be, in this case.)

Should Needle handle this case?

So the question that came to my mind was, “Should Needle handle this case?” Needle could, of course, be modified to check each newly instantiated service to see if it reponds to a special method name (like “initialize_service”), and if it does, invoke that method automatically. It could, but should it?

My initial urge was to say, “sure!” and jump in head first, coding like mad. This approach is, after all, supported by Copland and HiveMind. It works nicely in those frameworks. Upon reflection, though, I think it does not belong in Needle, and here’s why:

Copland and HiveMind try to be application frameworks. They try to do a lot of things, most of it magic, to make life easier for the developer. This makes those frameworks large and complex (which is not necessarily a bad thing).

The goal of Needle is to be a framework framework. That is to say, other frameworks may be built on top of Needle, adding complexity as needed. Needle itself should stay small, and relatively simple. This allows Needle to be fast as well, since the common case does not need (for example) to invoke an initialize_service method and should not be slowed down by an unnecessary call to responds_to?.

This means that developers using Needle directly may need to do a little more work (in some of the less common situations) than they would with other frameworks, like Copland. However, for most tasks, Needle will be at least as easy to use as those frameworks, and will scale much better, to boot.

If someone wanted to be able to add the automatic call to “initialize_service”, they could do it as a new service model. Or, as a new element in the instantiation pipeline, to use the terminology I described in Thoughts on Service Models.

So what approach will I take with the Transport::Session service? Not sure yet. It’ll be either the send-hash-to-constructor approach, or the setters-with-explicit-init approach. Tune in next time to hear the resolution to this dilemma…

This is the next article in a series about the process I am taking as I refactor the Net::SSH library to take advantage of dependency injection. In this installment, I’ll talk about what challenges I have faced as I’ve tried to refactor the “Key Exchange” algorithms, not only to make them “dependency injectable”, but also to make them unit-testable.

There are two different versions of the key exchange algorithm that are supported by Net::SSH. The refactoring process is very similar for both of them, so this will deal with the simpler of the two: Diffie-Hellman Group 1 SHA1.

Refactoring the Buffer Implementations

The original implementation had several explicit dependencies to the OpenSSL module, and part of the challenge was finding a good, clean way to decouple those dependencies. In particular, the algorithm relied heavily on the Net::SSH buffer implementation, which also had explicit dependencies on OpenSSL.

So, the first order of business was to refactor the buffer implementation. This was actually pretty straight-forward.

First, I removed the OpenSSL specific portions from the general buffer implementations, and put them in OpenSSL specializations of the buffer classes. Thus, I now had (for example) ReaderBuffer, and OSSL::ReaderBuffer, where OSSL::ReaderBuffer extended ReaderBuffer.

Then, I created a new factory service, “buffer_factory”, and a new “factory factory” for buffers (“buffer_factories”). The factory factory would return buffer factories depending on the cryptography backend in use, and the corresponding buffer factory then returned buffer implementations specific to the chosen cryptography backend.

Once the buffer implementation was refactored to my satisfaction, I turned my attention to the Diffie-Hellman algorithm.

Refactoring the Key Exchange

This was harder, becuase I had implemented the algorithm entirely in one method. Additionally, the algorithm depended on a “session” reference for sending and receiving messages (presumably over a network).

If my only goal was to make it “dependency injectable”, things would be simple. I merely had to create services for obtaining key and digest references, as well as for creating “bignum” instances appropriate for the cryptography backend in use. These all turned out to be trivially simple.

However, my secondary goal was to make this unit-testable. As it was, unit testing it was problematic, since it involved a short dialog with a remote server. To make it more feasible to test, I broke the algorithm into 6 discrete pieces, each of which I implemented in a separate, independent, publicly accessible method. Then, the orignal exchange_keys method just invoked those six methods, in order.

This allowed me to unit test each of those 6 pieces separately. It was still tricky-perhaps the trickiest bit of unit testing I’ve written to date, requiring more than a few mock objects to stand in for the dependencies-but it was at least possible. And for the first time, I feel confident in the key exchange code!