Net::SSH revisited

Posted by Jamis on July 29, 2007 @ 02:59 PM

I cringe whenever I remember those days, three years ago, when I was in the middle of a big Java project at BYU and was learning the ins-and-outs of dependency injection. For Java (and similarly constrained languages), DI is a technique that allows you to write modular code without tightly coupling the components. It seemed like a neat idea. So why do I cringe now?

Because I tried to bring that idea to Ruby, in the form of (first) Copland, and (later) Needle.

But hey, we’re all newbies at some time, right? I’ve since learned my lesson and have come to understand that although DI is a nifty technique that works well in some environments, it is wholly unnecessary in Ruby. Ruby sports an agility that Java will never know, and can get by without such crutches.

My conscience is salved somewhat by the knowledge that neither Copland nor Needle ever caught on. The only notable exception is the Net::SSH library, which I rewrote after RubyConf 2004 to take advantage of Needle, and which I originally held up as a shining example of how DI can help make code “more modular”. So, I cringe yet again, now, realizing that any time someone has installed Capistrano (which has become quite popular), they’ve also had to install Net::SSH, which carries the DI baggage along with it.

My shame has finally reached the pain point, though. I’m rewriting Net::SSH, taking advantage of many of the best-practices I’ve learned in the intervening years. It’s a lot slimmer now. Faster. Cleaner. Better. Even ignoring the ssh.shell and ssh.process services (which I’m not immediately porting to the new version), the code is less than half the size of the bloated DI-based version. It’s actually possible now to read and understand how the library works. And I’ve killed the use of threads to do polling and am doing it right, now, via IO.select. I’m really quite pleased about how it is turning out.

I’ve also managed to write a Net::SCP client library, which works quite well, especially considering that the SCP protocol (which is based on RCP) is apparently completely undocumented, and the canonical implementation from OpenSSH is…well…bad. Effective, but bad.

I’m going to be rewriting Net::SFTP next, which is another beast with its blatant abuses of metaprogramming. Once it is “done”, I’ll be packaging them all up as “version 2”, and releasing them aside a small update to Capistrano (tweaked as necessary to work with the minor API changes). I haven’t enjoyed working on Net::SSH this much in a couple of years! It really has been fun to gut it and rethink the architecture.

If you’re interested in following along at all, you can check out the Subversion repository:

And, after I start on the Net::SFTP rewrite, you’ll be able to follow it here:

The code is entirely undocumented, is lacking any tests at all, and there are a few small API changes that might bite you. Also, as I mentioned, ssh.shell and ssh.process are missing (they may be rereleased eventually as separate libraries). But, if you’re feeling brave and want to peek, I’d appreciate any feedback you might have.

Posted in Projects

Comments

Have something to add? Click here to leave a comment.

29 Jul 2007

1. Anko said...

Mind explaining in a couple of quick examples, why you don’t need DI in ruby?

Anko

2. Jamis said...

@Anko, primarily because the entire Ruby object space is one big DI container, and you inject dependencies by taking advantage of it. One reason people advocate using DI is to make testing each object in isolation easier. Well, you can do the same thing in Ruby by using mocking libraries (like mocha).

I find it ironic, that where once I had people asking me to give “a couple of quick examples” of why you need DI in Ruby, that now I’m being asked to justify not needing it. :)

3. john said...

More, please . . . would be nice to have some concrete code examples that show how di tripped you up and how you (un)refactored your code.

4. mla said...

I’d like some more details too. I’m new to Ruby and actually just learned the DI pattern and thought it was a good idea ;-)

The example I was shown was when a class needs a “logger” object. That you’d use DI to “inject” a logger object into the class from the larger environment, rather than having the object know how the create/configure the logger.

How would you do it without DI?

Thanks.

5. crayz said...

Take a look at the code in here: http://s3.amazonaws.com/harrisj-share/oscon2007.pdf

One thing it explains is how Rails Observers work, which should be architecturally the same as Loggers. It’s actually amazingly simple and beautiful

6. Jamis said...

This probably deserves a post on its own, but I’ll try and give a watered down justification here. If you’re interested, there’s a section “Beyond Java” by Bruce Tate which talks about why DI is not needed in Ruby, including quotes by yours truly, as well as Jim Weirich and DHH.

Ultimately, the fact that Ruby lets you reopen classes and redefine methods means that you can easily “inject” new references into your code at test-time. Consider the Logger example (which I find incredibly contrived, but the complexity of DI makes it hard to find a real-life bite-sized example that works). What’s wrong, really, with just doing this?

1
2
3
4
5
class Foo
  def initialize
    @logger = Logger.new
  end
end

Is it the explicit mention of “Logger” in the constructor that makes you nervous? If you want to make that more easily injectible, just move that to a factory method:

1
2
3
4
5
6
7
8
9
class Foo
  def initialize
    @logger = new_logger
  end

  def new_logger
    Logger.new
  end
end

When testing, then, you can reopen Foo and redefine new_logger. Or, even better, subclass Foo for testing and redefine new_logger. Or, even better, use a library like mocha that lets you override things like that on the fly!

Note that you can still use dependency injection techniques, in Ruby, too. When I say that I’m opposed to DI in Ruby, what I think I’m really saying is that I eschew specialized DI containers in Ruby. Factory methods, factory classes, blocks, metaprogramming—all of these can make a plain ol’ Ruby program extremely rich and powerful, without the need of something like Copland or Needle. And, when you DO need something more powerful, you can code it yourself, very easily. A simple Hash instance can be a powerful DI “container”.

Just don’t automatically start shopping for prepackaged DI solutions in Ruby. You just don’t need them. Show me an example where you think you do, and I’ll show you a very, very overarchitected piece of work.

7. Ryan Allen said...

Good ol’ metaprogramming. We need a book on the subject – or is there already one? Metaprograming ‘techniques’ I’ve learnt over the last 12 odd months is from tinkering with esoteric Ruby… I’ve tried to write a talk on it and found that there is really no concise way of teaching it without it coming across as a bunch weird ‘hacks’ that allow you to write really, really nice client code. Cheers for the code examples and explanation.

30 Jul 2007

8. malcontent said...

I would like to see Net::Rsync. There is a python version already.

9. mla said...

Thanks for the follow-up, Jamis. That makes a lot of sense.

Does anyone sort of freak out at the notion of redefining methods in arbitrary locations? Doesn’t that hint of “action at a distance” and raise the specter of horrible-to-find bugs?

And as an old-time Perl programmer, is this any different than how you redefine methods in Perl? Seems equivalent.

Thanks.

10. Jamis said...

@malcontent: sounds like you’re uniquely positioned to contribute something useful, then. :)

@mla: in practice, no, it doesn’t result in bugs that are hard to find. I’m not advocating redefining methods willy-nilly. In fact, I really only suggested doing it in in tests, which is a pretty limited environment. And I’m afraid I’ve only done the tiniest amount of Perl programming, so I can’t really answer your last question.

11. sri said...

i see that you indent some of the modules differently, so that they aren’t that deeply nested

module Net; module SSH; module Transport; module HMAC

but have you thought about putting end’s at the end of the terminating block. here is an example from capistrano:

    def load(*args, &block)
      options = args.last.is_a?(Hash) ? args.pop : {}
      args.each { |arg| load options.merge(:file => arg) }
      return unless args.empty?

      if block
        raise "loading a block requires 0 parameters" unless args.empty?
        load(options.merge(:proc => block))

      elsif options[:file]
        file = options[:file]
        unless file[0] == ?/
          load_paths.each do |path|
            if File.file?(File.join(path, file))
              file = File.join(path, file)
              break
            elsif File.file?(File.join(path, file) + ".rb")
              file = File.join(path, file + ".rb")
              break
            end
          end
        end

        load :string => File.read(file), :name => options[:name] || file

      elsif options[:string]
        instance_eval(options[:string], options[:name] || "<eval>")

      elsif options[:proc]
        instance_eval(&options[:proc])

      else
        raise ArgumentError, "don't know how to load #{options.inspect}" 
      end
    end

    def load(*args, &block)
      options = args.last.is_a?(Hash) ? args.pop : {}
      args.each { |arg| load options.merge(:file => arg) }
      return unless args.empty?

      if block
        raise "loading a block requires 0 parameters" unless args.empty?
        load(options.merge(:proc => block))

      elsif options[:file]
        file = options[:file]
        unless file[0] == ?/
          load_paths.each do |path|
            if File.file?(File.join(path, file))
              file = File.join(path, file)
              break
            elsif File.file?(File.join(path, file) + ".rb")
              file = File.join(path, file + ".rb")
              break end end end

        load :string => File.read(file), :name => options[:name] || file

      elsif options[:string]
        instance_eval(options[:string], options[:name] || "<eval>")

      elsif options[:proc]
        instance_eval(&options[:proc])

      else
        raise ArgumentError, "don't know how to load #{options.inspect}" end end

what do you think? i feel it much more readable.

12. Jamis said...

@sri, actually, I find that less readable, since you are now relying solely on the indentation to tell you where the block ends, which is one of the things I hate most about python. :)

13. James Hill said...

woohoo Net::SCP happy days, many thanks :)

31 Jul 2007

14. pauliephonic said...

Nice article, mainly as it’s good to see that a programming wizard such as your good self, needs to backtrack from time to time like the rest of us mortals :) P

15. sri said...

@Jamis thanks you for your comment. just wanted a long-time Ruby user’s opinion.

16. blackey said...

So i’m pretty new to ruby but i am in need of this scp script, how would you ‘install’ it to make it usable? For instance, i have scp.rb and download.rb and upload.rb in the same folder as my main.rb. in my main.rb i have require ‘scp’ and in my scp.rb i have require ‘download’ and require ‘upload’ but i still get an error.

Any help? Thanks!

01 Aug 2007

17. Neil Wilson said...

Excellent. I’m glad to see that the bloat caused by Java is finally being stripped back.

There is absolutely nothing wrong with being specific when you are using a dynamic language.

I thought in Ruby that the ‘initialize’ method was the factory for the class – since you can decide to call ‘new’ to create a new object, or ‘initialize’ directly to reinitialize an existing one.

18. Jamis said...

@blackey, alas it is not as simple as that. To actually use Net::SCP, you also need the new Net::SSH. You can grab all of the source code like so:

  svn export http://svn.jamisbuck.org/net-ssh/branches/v2 ssh
  svn export http://svn.jamisbuck.org/projects/net-scp scp

Do that in your project directory, so you now have “ssh” and “scp” directories there. Then, in your main project file, add the appropriate load paths:

1
2
$LOAD_PATH.unshift "path/to/ssh/lib", "path/to/scp/lib"
require 'net/scp'

From there, you’re on your own, though. I’ll focus on documentation once I’ve gotten everything working to my satisfaction.

02 Aug 2007

19. Nils Jonsson said...

I’ve been thinking similar thoughts about dependency injection frameworks in Ruby. They look good on paper but are overkill with this language.

Here’s a Ruby code snippet (with unit tests!) for enhancing attr_reader and attr_accessor. The upshot is a “just enough” solution for DI: http://snippets.dzone.com/posts/show/4382 .

20. Jamis said...

Nils, excellent stuff! I’ve wanted that functionality before. Really handy. However, it doesn’t appear to be related to dependency injection, but rather is just about specifying default values for attributes. Am I missing something?

07 Aug 2007

21. Zach said...

If you have a global logger why would you want to have to reopen and declare a method for each class that needed access to a logger?

To simplify things you would most likely use modules as mixins which would achieve the same effect as using a declarative style DI approach (much like http://injection.rubyforge.org uses) where you say “inject :logger” or “constructor :logger”, but even then you are still applying the same technique to your code just using a different mechanism.

Going the non-DI route do you see it making your tests harder to read or potentially more unmanageable or error prone? going non-DI you would end up with either subclassing in your tests, overriding methods on your actual object or partial mocking.

If I am testing an instance of MyClass it seems to be a cleaner approach to not have to touch it’s implementation in my test. And partial mocking seems to be something easy to abuse as you generally don’t want to mock out the object you’re testing.

Thoughts?

22. Jamis said...

Zach, if you have a global logger, then pass it as a parameter to the constructor. I find hash parameters are great for this kind of thing:

1
2
3
def initialize(options={})
  @logger = options[:logger] || Logger.new
end

My backlash against DI is really more of a backlash against DI frameworks. Tricks like the above give you a lot of the power of DI, without all of the baggage that something like Needle or Copland bring.

28 Aug 2007

23. Calvin said...

Does anyone have any clue what could be causing this error? /var/lib/gems/1.8/gems/net-ssh-1.1.2/lib/net/ssh/transport/session.rb:274:in `wait_for_message’: disconnected: Protocol error: expected packet type 30, got 34 (2) (Net::SSH::Transport::Disconnect)

Im trying to log into a Mikrotik router and without fail it gives me this error. Ive looked everywhere and cant find a thing about this error. Any ideas? Please e-mail

Calvin

29 Aug 2007

24. Jamis said...

Calvin, this post is probably the wrong place to be asking for support on Net::SSH 1.1.2, but it looks like your router is doing something funky. Packet type 30 is KEXDH_INIT, which the diffie-hellman-group1-sha1 kex algorithm uses to initiate a key exchange, and 34 is probably KEXDH_GEX_REQUEST, which the diffie-hellman-group-exchange-sha1 algorithm uses to request the key parameters…but 34 is one that the client sends to the server, not the other way around. If you enable verbose output and email me the full dump, I’ll take a look at it.

25. Calvin said...

Hello Jamis, That was the fix, Net::SSH.start(host, user, pass, :kex => ‘diffie-hellman-group1-sha1’) Logged right in! Thanks!