the all-thing | Thu Mar 11 13:11:09 -0500 2010
===============================================
Trollop 1.15 released
---------------------
Date: September 30, 2009 6:59pm
Author: William Morgan
Labels: trollop, releases
URL: http://all-thing.net/trollop-1.15-released.txt
I've just released Trollop [1] 1.15, which fixes an irritating misfeature
pointed out by Rafael Sevilla: when Trollop runs out of characters when it's
generating short option names, e.g. when you have a lot of options, it
shouldn't throw an exception and die. It should just continue peacefully.
Trollop's reign of domination [2] continues!
[1] http://trollop.rubyforge.org
[2] http://stackoverflow.com/questions/897630/really-cheap-command-line-option-parsing-in-ruby/1012930#1012930
(Leave a comment on this article at http://all-thing.net/trollop-1.15-released.txt.)
Ruby, Ncurses and blocked threads
---------------------------------
Date: August 6, 2009 6:40pm
Author: William Morgan
Labels: ruby, ncurses, sup
URL: http://all-thing.net/ruby-ncurses-and-thread-blocking.txt
If you're writing a multithreaded Ruby program that uses ncurses, you might be
curious why program stops running when you call @Ncurses.getch@. Sup [1] has
been plagued by this issue since 2005. Thankfully, I think I finally
understand it.
The problem is that there is a bug in the Ruby ncurses library such that using
blocking input will block *all* Ruby threads when it waits for user input,
instead of just the calling thread. So @Ncurses.getch@ will cause everything
to grind to a halt. This is probably due to the library not releasing the GVL
when blocking on stdin.
This bug is present in the latest rubygems version of curses, 0.9.1. It has
been fixed in the latest libncurses-ruby Debian packages (1.1-3).
To see if you have a buggy, blocking version of the ruby ncurses library, run
this program:
require 'rubygems'
require 'ncurses'
require 'thread'
Ncurses.initscr
Ncurses.noecho
Ncurses.cbreak
Ncurses.curs_set 0
Thread.new do
sleep 0.1
Ncurses.stdscr.mvaddstr 0, 0, "library is GOOD."
end
begin
Ncurses.stdscr.mvaddstr 0, 0, "library is BAD."
Ncurses.getch
ensure
Ncurses.curs_set 1
Ncurses.endwin
puts "bye"
end
(I purposely require @rubygems@ in there to load the rubygems ncurses library
if it's present; you can drop this if you don't use rubygems.)
There are two workarounds to this problem. First, you can simply tell ncurses
to use nonblocking input:
Ncurses.nodelay Ncurses.stdscr, true
But if you're writing a multithreaded app, you probably aren't interested in
nonblocking input, unless you want a nasty polling loop.
The better choice is to add a call to @IO.select@ before @getch@, which will
block the calling thread until there's an actual keypress, and then allow
@getch@ to pick it up:
if IO.select [$stdin], nil, nil, 1
Ncurses.getch
end
@IO.select@ requires a delay, so you'll have to handle the periodic nils that
generates. But the background threads should no longer block.
There is one further complication, which is that you won't be able to receive
the pseudo-keypresses Ncurses emits when the terminal size changes, since they
don't show up on @$stdin@ and thus the @select@ won't pass. The solution is to
install your own signal handler:
trap("WINCH") { ... handle sigwinch ... }
You will still see the resize events coming from @getch@, but only once the
user presses a key. You can drop them at this point.
That should be enough to make any multithreaded Ruby ncurses app able
function. Of course, once everyone's using a fixed version fo the ncurses
libraries, you can do away with the @select@ and set @nodelay@ to false.
(One last hint for the future: I've found it necessary to set it to false
before every call to @getch@; otherwise a ctrl-c will magically change it back
to nonblocking mode. Not sure why.)
[1] http://sup.rubyforge.org
(Three comments on this article at http://all-thing.net/ruby-ncurses-and-thread-blocking.txt.)
git wtf bf06ab7 released
------------------------
Date: July 28, 2009 8:13pm
Author: William Morgan
Labels: releases, git, git-wtf
URL: http://all-thing.net/git-wtf-bf06ab7-released.txt
I've released git-wtf version bf06ab7. The highlight of this release is
colorized output. ANSI escape sequences are the future of the web.
Also, the feature / integration branch comparisons is now only displayed when
@-r@ is supplied.
Check out the git-wtf home page [1] for an example of the fancy colorization,
or just download it now [2].
[1] http://git-wt-commit.rubyforge.org/#git-wtf
[2] http://git-wt-commit.rubyforge.org/git-wtf
(Four comments on this article at http://all-thing.net/git-wtf-bf06ab7-released.txt.)
My little continuation framework
--------------------------------
Date: June 18, 2009 9:26pm
Author: William Morgan
Labels: continuations, web
URL: http://all-thing.net/my-little-continuation-framework.txt
Since continuations are so fast in Ruby 1.9 [1], I've been playing with a
simple little contination-based web framework I wrote based on code I ripped
out of Whisper [2].
Using continuations is really awesome for webapps because you can write
application control flow just like you would write the regular program control
flow. You can write it in a single method, store state in local variables, use
@if@ statements, whatever.
For example, here's the entire code for a webapp that lets you increment and
decrement a number:
def run
counter = 0
while true
click = html <Current counter value: #{counter}.
To increment, #{link_to "click here", :increment}
To decrement, #{link_to "click here", :decrement}
Thanks!
EOS
case click
when :increment
counter += 1
when :decrement
counter -= 1
end
end
end
Pretty obvious huh? It's a loop, and when someone clicks on the increment
link, you increment the counter, and when they click on the decrement counter,
you decrement it. The app displays the HTML you see, and provides
@/increment@ and @/decrement@ links. The logic isn't spread out across
different methods and different classes like it would be in a regular
framework.
I added back-button support too, so pressing the back button after clicking
increment brings you to the previous number, and further increments and
decrements do the right thing.
I realize frameworks like Seaside and Wee have been doing this forever, but
it's new to me and very fun.
[1] http://all-thing.net/fibers-via-continuations
[2] http://masanjin.net/whisper/
(Leave a comment on this article at http://all-thing.net/my-little-continuation-framework.txt.)
Ritex 0.3 released
------------------
Date: June 18, 2009 2:41am
Author: William Morgan
Labels: ritex, releases
URL: http://all-thing.net/ritex-0.3.txt
I've released Ritex 0.3. No API or functionality changes; this is just a set
of miscellaneous tweaks that make Ritex work on Ruby 1.9.
(Leave a comment on this article at http://all-thing.net/ritex-0.3.txt.)
Fibers via Continuations
------------------------
Date: June 1, 2009 2:47pm
Author: William Morgan
Labels: ruby, ruby1.9, fibers, continuations
URL: http://all-thing.net/fibers-via-continuations.txt
In the last post I talked about some differences between fibers and
continuations [1]. What may not have been clear is that continuations are more
primitive and flexible than fibers are. In fact, you can implement fibers
using continuations.
Here's how. The basic idea is that we want to maintain two variables with
continuations in them, @inside@ and @outside@. The first one will transfer
execution into the block of code that forms the fiber. The second will
transfer control back to the outside world.
When the outside world calls @#resume@, we save our continuation point as
@outside@, and call the current @inside@ continuation. When, within the block,
@#yield@ is called, we save our current continuation point as @inside@, and
transfer code back to the current @outside@.
There are a few more details in terms of passing values from @#yield@ to
@#resume@, handling the return value of the block, and handling excessive
calls to #@resume@, but that's the basic story. Here's the code:
require 'continuation'
class CFiber
class Error < StandardError; end
def initialize &block
@block = block
callcc do |cc|
@inside = cc
return
end
@var = @block.call self
@inside = nil
@outside.call
end
def resume
raise Error, "dead cfiber called!" unless @inside
callcc do |cc|
@outside = cc
@inside.call
end
@var
end
def yield var
callcc do |cc|
@var = var
@inside = cc
@outside.call
end
end
end
This is also runnable on Ruby 1.8--just remove the @require@.
So why does Ruby 1.9 bother to implement fibers, when we can just use
continuations? I don't know what the real answer is, but "speed" is at least
a _good_ answer. Let's do some some benchmarking to compare the two:
require 'benchmark'
n = ARGV.shift.to_i
Benchmark.bm do |bm|
bm.report " fibers" do
f = Fiber.new do
x, y = 0, 1
loop do
Fiber.yield y
x, y = y, x + y
end
end
n.times { |i| f.resume }
end
bm.report "cfibers" do
f = CFiber.new do |c|
x, y = 0, 1
loop do
c.yield y
x, y = y, x + y
end
end
n.times { |i| f.resume }
end
end
We'll start with backporting that code to the Ruby 1.8.7 that Ubuntu provides
(ruby 1.8.7 (2008-08-11 patchlevel 72)). For 10000 Fibonacci numbers, we see:
table{margin-left: auto; margin-right: auto;}. | |_. user |_. system
|_. total |_. real | | _cfibers_ |>. 0.810000 |>. 0.070000 |>. 0.880000 |>.
0.879930 |
That's roughly 11.4kfps (that's thousand Fibonacci numbers per second) that we
can produce using continuation-based fibers.
Let's try the ancient Ruby 1.9.0 that Ubuntu provides (Ruby 1.9.0 (2008-06-20
revision 17482)):
table{margin-left: auto; margin-right: auto;}. | |_. user |_. system
|_. total |_. real | | _fibers_ |>. 0.040000 |>. 0.000000 |>. 0.040000
|>. 0.037583 | | _cfibers_ |>. 18.680000 |>. 1.770000 |>. 20.450000 |>.
20.482006 |
Wow, fibers are fast: 250kfps. But things have gotten significantly worse for
cfibers, clocking at a measely 0.489kfps for cfibers.
Finally let's try the latest and greatest Ruby 1.9.1 (ruby 1.9.1p129
(2009-05-12 revision 23412)):
table{margin-left: auto; margin-right: auto;}. | |_. user |_. system
|_. total |_. real | | _fibers_ |>. 0.040000 |>. 0.000000 |>. 0.040000
|>. 0.035148 | | _cfibers_ |>. 0.150000 |>. 0.000000 |>. 0.150000 |>.
0.155890 |
Fibers are just as fast as before, but continuations have improved
dramatically--from 11.4kfps to 66.6kfps. Still, native fibers are more than
three times faster.
So perhaps Ruby 1.9.1 is the best of both worlds. When you need fast
non-preemptive concurrency, you can use native fibers; when you need to
implement your own crazy control structures, you can use continuations and be
assured that they're still pretty darn fast (at least, as far as Ruby
operations are concerned).
[1] http://all-thing.net/fibers
(Leave a comment on this article at http://all-thing.net/fibers-via-continuations.txt.)
Fibers vs Continuations
-----------------------
Date: May 31, 2009 9:20pm
Author: William Morgan
Labels: ruby1.9, ruby, fibers, continuations
URL: http://all-thing.net/fibers.txt
Ruby 1.9 has both fibers and continuations. The two are often mentioned in the
same breath. They do vaguely similar-sounding things, and are implemented in
Ruby 1.9 with similar mechanics underneath the hood [1], much as how
continuations and threads were implemented with the same underlying mechanics
in Ruby 1.8 [2] [PDF, p. 14].
But implementation similarities aside, continuations and fibers have very
different semantics. A fiber behaves as a thread without preemption. Like a
thread, you create it, and it eventually dies; unlike a thread, you must
manually call @yield@ and @resume@ to transfer control in and out of it,
instead of just letting the runtime call them for you whenever it feels like
it. Like a thread, when you resume a fiber, you have the same call stack and
heap state (local variables) as when you left.
What's nice about fibers is that, since you keep explicit control of the order
of execution, you can get thread-like behavior without all the hassle of
mutexes and synchronization. Of course you have to deal with the hassle of
ordering all your operations, but you at least have the option of avoiding the
fun race-condition game that always seems to crop up in threaded programming.
What about continuations? Instead of fibers' create, kill, yield, and resume
operations, a continuation only really has two operations: capture and resume.
A continuation is captured once, and may be resumed multiple times. When you
resume a continuation, the call stack is reverted to what it looked like when
it was captured, but the heap state stays the same. There's no exit point or
death for a continuation (at least until Ruby gets bounded continuations);
execution simply continues from the capture point.
What's nice about continuations is that you can use them to implement control
structures. Loops, exceptions, cross-procedure gotos... almost every control
structure you can come up with can be implemented with continuations. In
fact, you can implement fibers using continuations [3]!
Let's look at an example. Here's the fiber-based Fibonacci computation from
the InfoQ article on Fibers in Ruby 1.9 [4]
fib = Fiber.new do
x, y = 0, 1
loop do
Fiber.yield y
x, y = y, x + y
end
end
20.times { puts fib.resume }
Here we call @yield@ from within the fiber once we've computed a number, which
transfers control to the main function, and which prints out the number
yielded and then calls @resume@ to transfer control back to the fiber. A
thread version looks very similar:
require 'thread'
q = SizedQueue.new 1
fib = Thread.new do
x, y = 0, 1
loop do
q.push y
x, y = y, x + y
end
end
20.times { puts q.pop }
Since we don't have explicit control over the scheduling, we implicitly
scheduled the order of operations by using a synchronized @SizedQueue@ data
structure, which blocks the computation thread from computing a new number
until the printing thread is ready to receive it. (There are many ways we
could've accomplished this.)
Here's the version using continuations:
require 'continuation'
c, x, y, i = callcc { |cc| [cc, 0, 1, 1] }
puts y
c.call c, y, x + y, i + 1 if i < 20
You'll notice there are no loops, and variables are never changed after
assignment. In fact the code is starting to look suspiciously like an
inductive proof, with one line that like a base case and another line that
looks like a recursive case. You can see why continuations make
functional-programming enthusiasts get excited!
This implementation works because resuming the continuation (the call to
@c.call@) replaces the call stack and point of execution with what they were
at the point it was captured (the call to @callcc@). In contrast, @resume@-ing
the fiber moved us back to the point we were when the fiber called @yield@,
and so the outer @loop@ in the fiber implementation was necessary.
Beyond call stacks, another major difference between fibers and continuations
is the way the heap is treated. Multiple fibers on the same section of code do
not share local variables. Multiple continuations on the same section of code
_do_. Here's a brief example. First, the fibers version:
fib = (0 ... 5).map do |i|
Fiber.new do
x = 0
Fiber.yield x
x += 1
end
end
fib.each { |f| puts f.resume }
We create five fibers, and call @resume@ on them once each. As you'd expect,
this prints out a series of 0's. The variable @x@ is not shared between the
multiple fibers. Of course, the fiber constructor here is a block, and blocks
are closures, so we could _make_ them share state by moving the @x = 0@ line
outside the @map@ line. But that's a result of having closures, not of fibers
per se.
Let's try an example with multiple continuations, all jumping into the same
point in the code:
require 'continuation'
x = 0
c = callcc { |cc| cc }
d = callcc { |cc| cc } if c
e = callcc { |cc| cc } if c && d
f = callcc { |cc| cc } if c && d && e
x += 1
puts x
c.call if c
d.call if d
e.call if e
f.call if f
We initialize @x@ to 0, create 4 separate continuations, add one to @x@, and
call the continuations in order. (The postfix @if@ statements ensure that the
continuations variables aren't set or called more than once. Calling @c.call@
without arguments will jump back to the @c = callcc@ line and set @c@ to
@nil@.)
Silly, but it illustrates the point: the output is "1 2 3 4 5", meaning that
the four continuations all share the same heap. When @d@ is called, its @x@ is
the same as the @x@ of @c@, and even though it was 0 when @d@ was captured, it
has since been modified by the resumption of @c@. When @e@ is called, its @x@
is also the same @x@, and so on. (In fact this whole example depends on this
behavior--each of the continuation variables are only set once, and must
"retain" their value across all rentries to continuations above them.)
In additon to multiple continuations being able to share state, the converse
is true too: multiple resumes on the same continuation will share state:
require 'continuation'
x = 0
c = callcc { |cc| cc }
x += 1
puts x
c.call c while x < 5
This outputs the same thing as the examples above.
Hopefully that clears up some of the confusion. Here's the summary:
table{margin-left: auto; margin-right: auto;}. |_. Fibers |_.
Continuations | |^. Four operations: create, exit, yield, resume. |^.
Two operations: capture and resume. | |^. Upon resume, call stack is wherever
it was at the last yield. |^. Upon resume, call stack is where it was when
captured. | |^. Do not share state except via closure. |^. Multiple
continuations and multiple invocations of the same continuation can share
state. |
[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/19609
[2] http://www.atdot.net/~ko1/pub/ContinuationFest-ruby.pdf
[3] http://all-thing.net/fibers-via-continuations
[4] http://www.infoq.com/news/2007/08/ruby-1-9-fibers:
(One comment on this article at http://all-thing.net/fibers.txt.)
Whisper 0.5 released
--------------------
Date: May 20, 2009 4:48pm
Author: William Morgan
Labels: whisper, releases
URL: http://all-thing.net/whisper-0.5.txt
I've released Whisper [1] version 0.5. Lots of good stuff since 0.3 (I didn't
announce 0.4 because it was a minor bugfix release):
* Nested comments are now properly supported.
* New and blocks added.
* A new @whisper-process-email@ command for manually reprocessing email. You
can also offload all email processing to this program instead of the main
Whisper server, if you like.
* New dependency for the 0.2 version of RiTeX [2], which has equation array
support (see announcement [3] for details).
* Better mbox-splitting code, now that I've figured out how to do this properly
in Sup [4].
* RiTeX macros now properly persist throughout an entry.
* Many other minor bugfixes: attribution lines in emails, various incorrect
bits of HTML output, escaping of Ritex error messages, etc.
Try it now!
1. @sudo gem install whisper --source http://masanjin.net/@
2. @whisper-init @
3. Follow the instructions.
[1] http://masanjin.net/whisper/
[2] http://ritex.rubyforge.org/
[3] http://all-thing.net/ritex-0.2
[4] http://sup.rubyforge.org/
(One comment on this article at http://all-thing.net/whisper-0.5.txt.)
Teaching your grandmother to suck eggs
--------------------------------------
Date: May 6, 2009 1:31pm
Author: William Morgan
Labels: ancient-greek, whisper, ubuntu
URL: http://all-thing.net/suck-eggs.txt
If you're reading a random diatribe on whether C and C++ are good for
numerical computing [1] and happen to come across the curious expression
"teaching your grandmother to suck eggs", and decide to learn more about it,
you'll quickly find references to early usages in the 1749 Henry Fielding
novel, ??Tom Jones [2]??, in which the protagonist recounts:
"I remember my old schoolmaster, who was a prodigious great scholar,
used often to say, Polly matete cry town is my daskalon. The English
of which, he told us, was, That a child may sometimes teach his
grandmother to suck eggs."
And if you then think to yourself, what the heck is "Polly matete cry town is
my daskalon"? you need only grab your handy copy of William Shepard Walsh's
1909 Handy-book of literary curiosities [3], look up "Polly matete" in the
index, and find that it's the transliteration (transphoneticization?) of:
"πολλοι μαθηται κρειττονες
διδασκαλον"
which is the last line of a Greek epigram attributed "sometimes to Phillippus
of Thessalonica, sometimes to Lucilius (both of whom lived in the early days
of the Roman Empire)", translated as:
Hermes, the volatile, Arcady's president,
Lacquey of deities, robber of herds,
In this gymnasium constantly resident,
Light-fingered Aulus bore off with these words:
Many a scholar, by travelling faster
On learning's high-road, runs away with his master.
So there you go. And if you're wondering what the original phrase means, Walsh
provides this helpful explanatory rhyme:
Teach not a parent's mother to extract
The embryo juices of an egg by suction:
The good old lady can the feat enact
Quite irrespective of your kind instruction.
As a side note, Whisper [4] now supports poems, and I just learned how to type
Greek in Ubuntu.
[1] http://scienceblogs.com/goodmath/2006/11/the_c_is_efficient_language_fa.php
[2] http://en.wikipedia.org/wiki/The_History_of_Tom_Jones,_a_Foundling
[3] http://books.google.com/books?id=hrJkAAAAMAAJ
[4] http://masanjin.net/whisper/
(One comment on this article at http://all-thing.net/suck-eggs.txt.)
Jaunty Jackalope rockin' my world
---------------------------------
Date: May 1, 2009 4:01am
Author: William Morgan
Labels: ubuntu
URL: http://all-thing.net/jaunty.txt
Ubuntu 9.04 is really pleasing me. Bootup time is 20 seconds (!) from GRUB to
login prompt, which is pretty incredible. It takes another 20 seconds from
typing my password and hitting enter to a Desktop with all my bells and
whistles loaded. In total, only 4 times the amount of time it takes for @ri
clone@ to print something useless!
Other things I'm enjoying:
* The fancy notification for IMs, wireless connections, etc is really sweet.
It looks nice, is non-intrusive, and properly stops when the corresponding
window has focus. I just wish it were somehow configurable.
* The orange satellite light on my laptop lights up every time the wireless is
accessed. Every time I send an IM I feel like I'm a fucking astronaut.
* Various wireless issues I was having with 8.10 seem to have disappeared.
* Screen profiles [1]. So cool.
[1] https://help.ubuntu.com/9.04/serverguide/C/screen-profiles.html
(Leave a comment on this article at http://all-thing.net/jaunty.txt.)
Pages
-----
* Page 1: You're reading it.
* Page 2: http://all-thing.net/index/1.txt
* Page 3: http://all-thing.net/index/2.txt
* Page 4: http://all-thing.net/index/3.txt
* Page 5: http://all-thing.net/index/4.txt
* Page 6: http://all-thing.net/index/5.txt
* Page 7: http://all-thing.net/index/6.txt
This delicious text version served up by Whisper .