In-case you haven’t done much work with EventMachine, it provides a mixin (called a Deferrable) that allows the developer to model their activity in the reactor loop as callbacks (success and failure). Using this pattern, we can create a class to register pieces of work and unregister them when the work is complete. In this way, we can make assertions on whether or not it is safe to shut down a process or not depending or not if we have work left to complete.

The exact situation I am trying to solve for is the following: I have a client that is pulling messages off of a queue and sending messages to another system (via another protocol). At any given time, we will have N number of IO jobs active in the event loop. But what happens when we send -TERM to this process? EventMachine starts closing these open connections and network calls that we want to succeed will fail. What I wanted was a way to register individual pieces of work (pulling the message off the queue and pushing to the remote service) and check to see if all the work is done (thus safe to allow the process to exit).

So consider the following class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class DeferrableProgressMonitor
  attr_reader :total

  def initialize
    @marked, @total = [], 0
  end

  def register(deferrable)
    @total += 1
    @marked << deferrable

    deferrable.callback { |object| @marked.pop }
    deferrable.errback { |e, object| @marked.pop }

    nil
  end
  alias :<< :register

  def current
    @marked.size
  end

  def wait?
    @total > 0 && !@marked.empty?
  end
end

Here is a small piece of code that uses it:

1
2
3
4
5
6
7
8
9
10

EM.run do
  monitor = DeferrableProgressMonitor.new
  trap("TERM") do
    unless monitor.wait?
      EM.stop
    end
  end

  monitor << thing_that_returns_a_deferrable
end

So the signal trap will make a call to check to see if there is any outstanding work left. if there is, we keep on going (my actual code sets a timer that continually keeps checking #wait?). Otherwise, we shut the loop down.

The are a few gotchas here:

• The signature for Deferrable#callback and Deferrable#errback will fail if it doesn’t match the call to Deferrable#set_deferred_status. So this pattern breaks down a bit as a general solution but works well if you have a well-defined activity whose completion is critical.
• Be mindful handling errors, anything that raises can cause you counters to be off, thus preventing the process without sending -KILL to it. You would want to try to catch what you can and utilize Deferrable#set_deferred_status to control success/failure.

a.ppend.to’s description:

Microblogging? How about Macroblogging? Use the power of Twitter to create a media-rich personal blog from just 140 characters of text. Simply send an @reply via Twitter to us (@ppend) and we’ll do the rest. We recognize different types of content including photos, videos, mp3 audio, quotes, and more - so start posting!

Please check it out when it goes to public voting next week and vote for us on our team page.

I wound up stumbling upon Brian Lopez’s yajl-ruby which is a ruby binding for a native streaming json library. It has excellent support for dealing with streams and also supports persistent http connections (perfect for the streaming API).

Below is some code utilizing yajl that opens a connection and pushes the screen name and text of a tweet to the console as it is received. (disclaimer: This is not even close to being production-ready as Twitter has strong recommendations for reconnect logic and dealing with the API’s alpha QOS). This is using the publicly available sprinkler stream; which is only a subsection of the public timeline but should be good enough for general application use (there are gardenhose and firehose streams available for statistical purposes but access to these is at Twitter’s discretion).

1
2
3
4
5
6
7
8
9
10
11
12
13
14

require "uri"
require "yajl/http_stream"

u = "twitter username here"
p = "twitter password here"

url = URI.parse("http://#{u}:#{p}@stream.twitter.com/spritzer.json")
Yajl::HttpStream.get(url, :symbolize_keys => true) do |status|
  unless status.has_key?(:delete)
    puts "screen_name: #{status[:user][:screen_name]}"
    puts "message: #{status[:text]}"
    puts
  end
end

And that is it. I can point out that the streams will additionally push delete operations out that are to be honored by client applications. Above I am filtering out the delete messages by key name and only printing out actual statuses.