I’ve got lots of code to share with you all today:

Net::SSH 2.0 Preview Release #3

Not very many external changes, but channels now have an on_open_failure callback that you can employ to be told when a channel could not be opened.

1
2
3
4
5
6
7

channel = ssh.open_channel do |ch|
  # ...
end

channel.on_open_failure do |ch, code, reason|
  puts "could not open channel because #{reason.inspect} #{code}"
end

This change was necessary to make port forwarding a bit more reliable for Net::SSH::Gateway (see below).

Also, individual Net::SSH sessions may contain their own custom properties, much as Net::SSH channel objects do, which can be quite handy when you need to keep some state associated with the session.

ssh[:key] = "value"

This change (and several other internal refactorings) was necessary for the correct working of the Net::SSH::Multi library (see below).

To get the preview release #3 (version 1.99.2):

gem install --source http://gems.jamisbuck.org net-ssh

You can follow development via GitHub:

http://github.com/jamis/net-ssh/tree/master

Net::SFTP 2.0 Preview Release #2

All that was added in this preview release was custom properties on Upload instances:

1
2
3

uploader = sftp.upload("local.file", "remote.file")
uploader[:failed] = false
# ...

This was necessary to get Capistrano file transfers to work with Net::SFTP 2.0.

To get the preview release #2 (version 1.99.1):

gem install --source http://gems.jamisbuck.org net-sftp

You can follow development via GitHub:

http://github.com/jamis/net-sftp/tree/master

Net::SSH::Gateway 1.0 Preview Release #1

Net::SSH::Gateway is (in essence) the extraction of the gateway code from Capistrano into its own library. It lets you tunnel SSH connections through some “gateway” server to servers that would be otherwise inaccessible.

1
2
3
4
5
6

require 'net/ssh/gateway'

gateway = Net::SSH::Gateway.new('gateway.host', 'username')
gateway.ssh('remote.host', 'user') do |ssh|
  # ...
end

You can also use it as a general facilitator for forwarding connections over a local port, when you don’t care what port is to be used—you just want to connect to a remote server.

1
2
3

gateway.open('remote.host', 80) do |port|
  Net::HTTP.get_print '127.0.0.1', '/path', port
end

The current thinking is that the next release of Capistrano will ditch its own gateway implementation in favor of Net::SSH::Gateway. To get this first preview release (version 0.99.0):

gem install --source http://gems.jamisbuck.org net-ssh-gateway

You can follow development via GitHub:

http://github.com/jamis/net-ssh-gateway/tree/master

Net::SSH::Multi 1.0 Preview Release #1

Net::SSH::Multi is the guts of Capistrano, extracted into a library of their own. It allows you to define and categorize servers, and then execute commands in parallel on them, or on subsets of them, using an interface similar, if not identical, to that of Net::SSH::Connection::Session and Channel.

Eventually, Capistrano will be refactored to take advantage of Net::SSH::Multi, but doing so will require some significant changes to Capistrano’s innards, and would almost certainly break many third-party libraries. Thus, you won’t see Capistrano on Net::SSH::Multi until Capistrano 3.0 or so (which will not be the next release of Capistrano). However, you can use Net::SSH::Multi to implement most of Capistrano’s functionality in whatever form you like. Want Capistrano-in-a-rakefile? Have at it!

1
2
3
4
5
6
7
8
9

require 'net/ssh/multi'

Net::SSH::Multi.start do |session|
  session.use 'jamis@somewhere.com'
  session.use 'jamis@elsewhere.com'

  session.exec 'hostname'
  session.loop
end

Grab the first preview release (version 0.99.0):

gem install --source http://gems.jamisbuck.org net-ssh-multi

You can follow development via GitHub:

http://github.com/jamis/net-ssh-multi/tree/master

For all of these libraries I’ve tried to make the rdoc and ri documentation as informative as possible, so please refer to those to get started. If those aren’t helpful enough, let me know what could make them more useful!

Net::SSH 1.x is thread-safe. Net::SSH v2, as currently implemented in its pre-release state, is not.

I’ve debated this long and hard with myself. Wrapping code in mutexes and doing all the other stuff that thread-safety requires adds a surprising amount of overhead, and it feels painful to me to have to add all that when the most common use-case of the library will be in single-threaded environments. However, without the mutexes and all that overhead, using Net::SSH in a multi-threaded environment (where multiple threads are hitting the same Net::SSH connection) will result in some rather ugly errors, and would require those programs to add their own mutexes and such to protect the integrity of the connection.

I’m leaning towards leaving the library optimized for the single-thread experience, and requiring the people wanting multiple threads of execution to fend for themselves. However, in the interest of getting feedback from people who might actually use the library, I ask you: which would you prefer? A faster library in single-threaded programs? Or a simple program in multi-threaded ones? Is there a general best-practice in this case?

I think...I think I’m actually, done, more or less, with Net::SFTP v2!

I just committed a last bit of documentation tweaking, and although I’m sure there’s more of that to come, and bugs to fix and features to add, I’m calling this bit “good enough”, and I’m moving on now to Net::SCP.

I’m not actually releasing Net::SFTP v2 at this point—it’s pretty much useless without a corresponding Net::SSH v2 release. So why am I saying anything about it right now? Because it’s the first significant milestone that I’ve completed in my track to complete the new Net::SSH suite of libraries! And I’m excited! And I’ve learned something significant by finishing Net::SFTP v2 before completely finishing Net::SSH v2: by completing a dependency of the primary library first, I saw first-hand where the deficiencies in the primary library were. This has resulted in a much tigher, much more useful API than the Net::SSH v1 and Net::SFTP v1 libraries.

Now, there is no real release of Net::SSH v2 and friends, but that’s not to say that if you’re brave, and don’t mind exploring a bit, you couldn’t get the new Net::SSH and Net::SFTP to work. The new Net::SSH is actually mostly done—I just need to revisit the documentation and update a few things here and there. The code is all on github. If you don’t have git, you can download a tarball directly from github, even.

So, have at it, if you’re so inclined. (And if you’d like on the github bandwagon, I’ve still got two invites burning a hole in my pocket…email me at (my first name) at jamisbuck.org, and I’ll be happy to send you one.)

Update: my github invites are all gone. Sorry!

I’ve got my plate full of projects that need attention, and the stress has been getting to me. Everybody has different ways of dealing with stress, but I think mine is the least effective: ignoring things doesn’t make them go away.

So, I’m taking a more active approach. Starting immediately, I’m going to be spending one hour, every day, six days a week, clearing away the backlog of projects. I’ll post a comment each day to keep myself honest—if I don’t leave a comment, you may assume I did not do any work that day!

Here is a list of what I’ll be working on, more or less in order of priority:

1. Capistrano 2.2.0. This will primarily be a maintenance release, and mostly requires that the patches that are pending be tested and applied. It will also update the Net::SSH and Net::SFTP gem dependencies so that they explicitly exclude any 2.0 versions of those gems.
2. Net::SFTP 1.1.1. This is a super minor release with no new features. All it will do is update the gem dependency on Net::SFTP such that it won’t try to load Net::SSH v2.
3. Net::SSH v2, Net::SFTP v2, Net::SCP v1. These will be released simultaneously. They are an 80% rewrite of the original code (and yeah, rewrites are evil, but sometimes they are necessary). The API has changed, and scripts written against the existing API will almost certainly be broken by the upgrade, so make sure you are depending on version 1.x.
4. Capistrano 2.3.0. This is another maintenance release, which simply modifies Capistrano to work with the new Net::SSH and Net::SFTP versions. (Possibly Net::SCP, too, not sure about that yet).
5. SQLite3/Ruby 1.2.2. A maintenance release. Haven’t seen a release on this for a long, long time. No new features planned, but lots of bug fixes are pending.

I’m not even going to try and give an ETA on these. Just know that I’m working through them, an hour a day, and that progress is finally being made. I’ll post what I worked on in the comments to this thread, for those who want to follow along.

I’m really pleased with how Net::SSH v2 is turning out. I’ve actually got it to a point where it’s about ready for a first preview release! If you’re an adventurous soul, you can grab the preview release gem (version 1.99.0) here:

• Net::SSH 2.0 preview #1 (1.99.0) gem

Note that Net::SSH v2 is not going to be 100% backwards compatible with 1.x. Many things will work mostly the same, but here are a few of the most significant gotchas:

• Net::SSH.start now uses a fixed parameter list, instead of the polymorphic monster it was. This makes it much easier to document, and use. The first parameter is always the host you are connecting to, and the second is always the username you’re connecting as. Options always come last.
• The shell, sync-shell, and open3 services have been dropped. You won’t find them in Net::SSH 2.0. If you want them, I strongly encourage you to write them and release them as third-party extensions to Net::SSH.

There are also a couple of (much needed!) short cuts for executing commands:

1
2
3
4
5
6

Net::SSH.start("localhost", "jamis") do |ssh|
  puts ssh.exec!("echo hello world!")

  ssh.exec("hello world")
  ssh.loop
end

The first, #exec!, will block until the command finishes, and will return the output. The second will return immediately, before executing the command, and will run when the event loop does, writing output to the terminal.

The documentation is pretty much done, too. To see it, just install the gem, change to the net-ssh-1.99.0 gem directory, and run “rake manual”. Then open “doc/out/index.html” in a browser. It includes a brief tutorial, a quick-reference, and an FAQ-style “how do I do X” interface.

So! Give it a try. Remember that the current release of Net::SFTP won’t work with this new version (at all), and neither will Capistrano. But the lovely thing about RubyGems is that you can easily install and uninstall, on demand.

Oh, and before I forget, a couple of little teasers about what’s coming up:

1
2
3
4
5
6
7
8
9
10

require 'net/scp'
Net::SCP.download("localhost", "jamis", "/Users/jamis/.bashrc", "./my-bash-rc") do |name, sent, total|
  if sent == 0
     print "downloading #{name} (#{total}b): "
  elsif sent == total
     puts
  else
     print "."
  end
end

and…

1
2
3
4

require 'uri/open-scp'
open("scp://jamis@localhost/Users/jamis/.bashrc") do |io| 
  puts io.read 
end

I cringe whenever I remember those days, three years ago, when I was in the middle of a big Java project at BYU and was learning the ins-and-outs of dependency injection. For Java (and similarly constrained languages), DI is a technique that allows you to write modular code without tightly coupling the components. It seemed like a neat idea. So why do I cringe now?

Because I tried to bring that idea to Ruby, in the form of (first) Copland, and (later) Needle.

But hey, we’re all newbies at some time, right? I’ve since learned my lesson and have come to understand that although DI is a nifty technique that works well in some environments, it is wholly unnecessary in Ruby. Ruby sports an agility that Java will never know, and can get by without such crutches.

My conscience is salved somewhat by the knowledge that neither Copland nor Needle ever caught on. The only notable exception is the Net::SSH library, which I rewrote after RubyConf 2004 to take advantage of Needle, and which I originally held up as a shining example of how DI can help make code “more modular”. So, I cringe yet again, now, realizing that any time someone has installed Capistrano (which has become quite popular), they’ve also had to install Net::SSH, which carries the DI baggage along with it.

My shame has finally reached the pain point, though. I’m rewriting Net::SSH, taking advantage of many of the best-practices I’ve learned in the intervening years. It’s a lot slimmer now. Faster. Cleaner. Better. Even ignoring the ssh.shell and ssh.process services (which I’m not immediately porting to the new version), the code is less than half the size of the bloated DI-based version. It’s actually possible now to read and understand how the library works. And I’ve killed the use of threads to do polling and am doing it right, now, via IO.select. I’m really quite pleased about how it is turning out.

I’ve also managed to write a Net::SCP client library, which works quite well, especially considering that the SCP protocol (which is based on RCP) is apparently completely undocumented, and the canonical implementation from OpenSSH is…well…bad. Effective, but bad.

I’m going to be rewriting Net::SFTP next, which is another beast with its blatant abuses of metaprogramming. Once it is “done”, I’ll be packaging them all up as “version 2”, and releasing them aside a small update to Capistrano (tweaked as necessary to work with the minor API changes). I haven’t enjoyed working on Net::SSH this much in a couple of years! It really has been fun to gut it and rethink the architecture.

If you’re interested in following along at all, you can check out the Subversion repository:

• Net::SSH v2: http://svn.jamisbuck.org/net-ssh/branches/v2
• Net::SCP: http://svn.jamisbuck.org/projects/net-scp/trunk

And, after I start on the Net::SFTP rewrite, you’ll be able to follow it here:

• Net::SFTP v2: http://svn.jamisbuck.org/net-sftp/branches/v2

The code is entirely undocumented, is lacking any tests at all, and there are a few small API changes that might bite you. Also, as I mentioned, ssh.shell and ssh.process are missing (they may be rereleased eventually as separate libraries). But, if you’re feeling brave and want to peek, I’d appreciate any feedback you might have.

So, this was totally news to me, but apparently Mac OS X Leopard will bundle Capistrano. How about that! It’s probably too late for them to bundle cap2, even if I were to release it tomorrow, but at the very least this is motivating me to get things rolling again. I’ll be releasing the third (and final) preview release of Capistrano “soon”, and then the focus will be on getting the capify.org site polished enough for the final release on cap2. I’d expect to see cap2 final out late this month, or early July.

Someone, hold me to that!

I’ve been working on rewiring parts of Capistrano, and I’m making some good progress in a few different areas. One thing that has continued to defy me is actually unit testing Capistrano. I openly admit the existing unit tests for Capistrano are remarkably lame, but I’m at a loss as to how to improve them.

So, I’m opening the floor on this one. How would you do it? Specifically, allow me to point you to the gateway implementation, which uses Net::SSH and sockets, and which has proven particularly tricky for me to see my way through to unit testing. Other areas that I would appreciate suggestions are: what is a good way to go about testing the CLI (command-line interface) module? What would be a technique for actually testing the standard tasks? And how would you test the file transfer abstraction and the capistrano shell?

I know from experience that code which is hard to test is usually poorly designed, so I’m prepared to hear comments like “you’ve misdesigned this thing, you should have done X, Y, and Z.” Of course, the gentler you can break it to me, the happier I’ll be. ;)

So, tell me, how would you do it?

So, as I’ve mentioned a few times recently, I’ve been working on a logic engine for Ruby, using a Ruby-ish DSL. It’s working quite nicely now, and is able to do the examples that Mauricio so kindly provided, including the Tower of Hanoi solver, and the symbolic differentiator.

The exercise has been fun, as well as very educational. For the curious, you can check it out at http://svn.jamisbuck.org/projects/logic_engine. For now, I’d like to talk a bit about how I implemented the DSL, since that’s probably of more general interest.

→ Click here to read the rest of this article.

I’ve been asked several times recently if I had any plans to implement in Capistrano the things that Dave Thomas mentioned in his RailsConf keynote address. Not having been there, I had no idea what they were talking about! However, I finally made some time and watched the keynote.

First off, Dave brings up some really good points. The split between developer and sysadmin is a real thing, even if it is one that I don’t wholly agree with. Prerequisites and such are good ideas.

That said, I’m not going to implement them. Or rather, I’m not in a position to implement them right now.

The environments that Dave describes are not the environments I live and work in. If I were to try and implement those things, I’d just be guessing, and I’ve got a pretty bad track record when it comes to guessing what other people need. I’ve learned the hard way to write what I need, and only what I need—and Capistrano, as it stands, is pretty darn close to what I need.

That’s not to say that Capistrano is perfect. It certainly isn’t, and I’ve got plans for further improving it. (In fact, stay tuned for an announcement of a new release, coming up.) However, my vision for Capistrano is not Dave’s vision.

If Dave’s vision happens to be your vision, or if your vision for deployment otherwise differs from mine, then I have a challenge for you: write a deployment system that does what you need it to do. If there’s one thing I’ve learned from Rails, it is that extraction is the only True Way to developing a truly useful system. Perhaps you could be the author of the next generation of Capistrano!

Things are getting serious, now. I’m going to be co-authoring a book with Aaron Huslage, tenatively titled “Beyond Rails with SwitchTower”, to be published by Addison-Wesley. It’s the newest kid on the book-block, and (as far as I know) the first to be solely dedicated to SwitchTower.

Exciting stuff!

Well, I didn’t get nearly as far on SwitchTower over the weekend as I would have liked, but here’s what’s currently residing in the repository:

• Custom SSH connection options (so you can specify the port, and other things)
• The Rails rake tasks for SwitchTower now load ST directly, instead of invoking it via system. This should clear up issues with using the rake tasks on Windows.
• You can set the :checkout variable to "export", to use export instead of checkout when deploying. (Only supported by the subversion module, currently.)

I’m still planning on implementing the following features (which should all be available by the next release of SwitchTower):

• Verbosity settings in deploy.rb
• Deploy via copy instead of checkout/export
• Handle the SSH auth error that occurs when you don’t have the remote hosts in your .ssh/known_hosts yet
• Add a setting for run vs. sudo in the restart task
• Clean up old deployment directories

Lastly, I’m still considering what to do about the following requests. These may or may not find their way into the next release of ST. Some of them may never see the light of day in an official ST release.

• Optionally leave the failed deployment on the server if it fails (instead of deleting it, as currently happens.)
• Subclassing roles, so you can (for instance) have a specialized “app” role.
• Built-in support for staging servers
• Notification emails
• Allow different user/pass for SVN access than for SSH access
• Per-server deployment credentials/settings

I may try to come up with a plugin architecture (minimalistic, like Rails’ plugins) so that some of these can be addressed by third-parties in a modular fashion, but we’ll see how it all falls out.

Thanks for the suggestions, everyone!

SwitchTower? It seems to be popping up everywhere, with people generally raving about it. Some seem more than a little enamored of it.

I’ll be releasing a new version of it early next week.

But what I’m really wondering is this: what about SwitchTower really bothers you? What could make it better? What do you think it lacks? No guarantees that I’ll answer every request (in fact, I’d like to keep the features pretty limited, and allow people to instead distribute their own recipe files where possible), but I’d like to know how people are using (or wanting to use) SwitchTower, and what about it is most painful to them.

So, speak up!

Don’t write software you don’t need.

I mean it. Don’t do it. If you don’t need it right now, don’t write it. Don’t waste your time.

David is going to get a good laugh out of this article, because I’m only going to be parroting a lot of what he has been preaching for so long, about “frameworks are extractions.” I finally realized that he’s been talking to me. Every time he or Jason shouts “LESS SOFTWARE”, it’s been at me. I just didn’t realize it at the time.

If you write software you don’t need, you write software you won’t use.

If you write software you won’t use, you don’t know how the software will ultimately be used. You might think you have some idea. You might even have some concept of one or two practical use cases. But really all you’re doing is playing a “this might be a cool feature” game, and it’s not until the feature requests come rolling in that you realize you’ve missed the point.

And, if you’re anything like me, you’ve now got a sense of obligation to make your creation useful, despite your rapidly decreasing interest in the project. And at that point, the project ceases to be fun, and becomes a burden.

I’m guilty of this. Very, despicably guilty. Consider the following list of projects, and see if you can figure out what they have in common:

• Net::SSH
• Net::SFTP
• SQLite/Ruby
• SQLite3/Ruby
• Copland
• Needle
• Ruby/CAPTCHA

What? I wrote them? Well, yes, I did, but that wasn’t necessarily the commonality I was fishing for.

The common feature of all of them is that I didn’t need them when I wrote them. In fact, for most of them, I haven’t hardly used them since I wrote them. I don’t think I ever actually used Copland at all, nor the CAPTCHA. Net::SFTP was more a “logical next step” after Net::SSH. I wrote SQLite/Ruby because it seemed like a fun project at the time. I wrote SQLite3/Ruby because I wanted to experiment with SWIG.

Net::SSH, though, takes the cake. I wrote it because I thought that I might someday want to write a maven workalike for Ruby, and one of the features it sports is the ability to deploy libraries via ssh. Logically, I would need an SSH library in Ruby. It seemed like a good place to start at the time…

And Copland and Needle… well, suffice it to say that I’m not as convinced of the necessity of dependency injection in Ruby as I once was. A guy is entitled to change his opinions every once in while, right?

At any rate, I now have a few projects that I need, but due to this overpowering sense of obligation about the above list, I feel guilty every time I try to work on them. Instead, all I can focus on is how I ought to be documenting Net::SFTP, or addressing the bug and feature requests for both of the SQLite/Ruby projects. Or fixing a security concern about Ruby/CAPTCHA that was brought to my attention months ago.

Sigh.

So, I come to the point of this article: I’m looking for individuals that would be willing to adopt one or more of the projects listed above. I need to divest myself of them, and leave them in the hands of people that actually need them. This would benefit both the Ruby community (because there would be new life breathed into these projects) and myself (because I could focus on the software I need, again).

If you’re interested, let me know.

Update: Justin Dossey has volunteered for Ruby/CAPTCHA, so that’s one down. Thanks, Justin!

I honestly cannot remember being this busy for a long time.

Working to get Backpack ready has been really, really fun. I’ve been putting in more hours on this than I ever did on anything at BYU, and enjoying almost all of those hours immensely. (Ruby and Rails both have a great deal to do with that. Writing code in anything else is painfully tedious!)

However, for the first time in awhile I’ve found myself with some time. My wife has gone out to run some errands, the kids are sleeping, and I’ve just finished patching a few more Backpack bugs (yah, on a Saturday!). But what do I find? A whole slough of side projects languishing and hungry for some loving!

Specifically:

• Net::SSH. There are a few new bug fixes in the repository. I need to package release an update.
• Net::SFTP. Needs documenting badly. Once the docs are there, this is ready for a 1.0 release.
• Syntax. I fixed several bugs some time ago, and never got around to packaging a release. What I really need to do is rethink this library—the regex-based approach is simple, but also impossible to use for parsing every valid Ruby construct. Maybe its time to go with a real parser? (Yah, like I’ve got the time for that!)

Other unreleased projects that I’m working on:

• Ruby/TZ. A time-zone lib for Ruby, which will make it possible to support time zones and daylight-savings in a cross-platform manner.
• Budgetwise. A personal finance application, in rails, that makes budgets the central theme instead of a sideshow.

However, all is not well in paradise. Net::SSH and Net::SFTP are starting to annoy me. I don’t want to work on them any more, because the architecture feels too over-wrought and the interface too unplanned. Consider what a pain it is to simply execute a command and return it’s output, synchronously. (If you haven’t tried to do that yet in Net::SSH, it’s probably because you haven’t used Net::SSH. I mean, it’s got to be one of the most common use cases, and yet it’s incredibly difficult to do!) If I had it to do all over again (yah, it would be rewrite #3 of the Net::SSH stuff) I’d start with the interface and work backwards, an approach I’ve learned from 37signals that works equally well on library API’s as it does on user interfaces.

Syntax is starting to annoy me. As I said before, the regex-based parser (though fast, and 98% effective) is not a true parser. It can’t parse all valid Ruby constructs. Can’t, I say! No amount of patching will turn it into a true parser, and that knowledge makes me want to stop patching it altogether. So, if I had that to do all over again, I’d start with finding or writing a good, pure-Ruby parser and writing a syntax highlighter just for Ruby. Then, I’d do the same for HTML, YAML, and C (the other three syntaxes I do any amount of work with).

Ruby/TZ and Budgetwise are both fun projects, and I look forward to working on those, but I feel some obligation to get at least the Net::SSH and Net::SFTP libs current. Syntax, too, I guess…

I’m so conflicted! :)