Rethinking Sup part II
In Rethinking Sup part I, I concluded that Sup the MUA is an evolutionary dead end, and that the future lies in Sup the Service (STS). But what does that mean?
One thing I want to make clear it does not mean is any abandonment of the Sup curses UI. That particular “user experience” has been refined over the past few years to become my ideal email interface. It would be silly to throw that away.
What will happen to the curses code is that it will become one client among
(hopefully) many. Once there’s a clear delineation between UI and backend, you
can make a UI choice independent of making a choice to use Sup in the first
place. You can run sup-curses-client if you want. Or you can build
a web interface, or an Openmoko interface. Working with ncurses has always been
the least enjoyable part of Sup, so maybe I’ll actually enjoy learning
Javascript.
What backend functionality will STS actually provide? If I were simply reworking Sup into a client and a server, the obvious answer would be “a searchable, labelable, threaded view of large amounts of email”.
But reworking Sup is a great time to extend its original goals. In particular, I would love for STS to handle to other types of documents besides email. I’ve always used my inbox as a mechanism for writing notes to myself. I’ve experimented briefly with reading RSS feeds through it. I’d like STS to support email, of course, but not to be limited by it.
My grand vision: STS will be a searchable, labelable, threaded view of large numbers of documents.
You can throw whatever you want in there, and STS will store it, thread it, and let you label and search for it. Email, RSS feeds, notes, jabber and IRC logs, web pages, RI documents—I want you to be able to throw them all in there. I want you to be able to annotate any of those things by adding notes and threading them against the original objects. Basically I want STS to be the primary tool you use for organizing and recalling all the textual information you’ve ever encountered in your life.
Cool, huh?
There’s another convenient benefit to this transformation: no one will expect STS to act like a MUA. STS does its own storage. You add your email and your other documents to the server and then you can throw those files away (or not). There are no more questions of supporting IMAP or various mbox dialects or “why doesn’t Sup treat Maildir correctly”. The files are in STS, and once they’re their, they’re out of your hands. You’ll be able to export them, of course, and if you’re crazy you might be able to write an IMAP server translation layer for STS, but there will be no more expectation of realtime Maildir handling. As I explained in part I, that’s a game I don’t want to play.
STS is a grander vision than a MUA, and it no longer has to be hobbled by the constraints of being expected to act like one.
Some other nice benefits of reworking Sup into SYS:
- You’ll be able to run multiple clients at once.
- It’s an opportunity to rework some things. For example, one of the most noticeably slow operations in Sup (“Classic”) is assembling a large thread. This is because I made a decision early on to do all threading at search time. That made certain things easier (in particular, I could change the threading model without having to rescan the entire index), but in retrospect the cost is too high. STS will maintain document trees directly.
- I can replace Ferret with Sphinx. It’s been a good couple years, but the periodic non-deterministic index corruption that’s been an issue for over a year is an exit sign to me. Working with Sphinx is nowhere nearly as nice as working with Ferret, but speed and stability go a long way.
I’ve been working on the code for STS on and off for the past couple weeks and it’s slowly starting to come together. Once the major components have at least been all sketched, I will host a git repo.
Discussion
To leave a new comment, enter your email address. A copy of the article will be sent to you via email.
I believe this can be done without compromising the basic user experience, which I would be very reluctant to do because it has been lovingly tweaked over the years to be William’s Ideal Email Experience.
I love sup and I’m not bothered by any of the problems you describe so this whole thread just seems philosophical and worrisome to me. But I’m reassured that you think you can do whatever without breaking the Ideal Email Experience. Thanks again for sup!
I’m not sure not being able to move mails between stores and setting state is such a big deal. But what is a showstopper, is that sup probably doesn’t save the mails I send anywhere?
Anonymous: Sup does save a copy of your send emails. Look in ~/.sup/sent.mbox.
Moving mails around between mailstores and setting state is a frequent request, because people (rightly) expect Sup to behave like a regular MUA like mutt.
Furthermore, Sup is aimed at the mailstores of the future (my present mailstores), which are so big that mutt can’t handle them anyways.
Out of curiosity, what are your mailstores? I ask because I’ve had to abandon sup and go back to mutt, because anything over 50k messages, spread throughout several Maildir mailstores, caused a LOT of instability and weird behavior in sup, while mutt seems to handle everything without a hiccup.
Of course, I’m now growing ever-farther behind on keeping up with my mail, since mutt makes it too easy to ignore things, but I had to make the trade-off of “works” vs “global view”.
Out of curiosity, what are your mailstores? I ask because I’ve had to abandon sup and go back to mutt, because anything over 50k messages, spread throughout several Maildir mailstores, caused a LOT of instability and weird behavior in sup, while mutt seems to handle everything without a hiccup.
I’ve always used mbox files. The only instability I’ve seen is Ferret’s inevitable index corruption.
I don’t use Maildir, but I know a lot of people do. I’m surprised that you had so much trouble because AFAIK they are all happy with it.
At any rate, I am in the midst of writing “Sup the Service” which should fix these issues in that I will control how things are stored on disk. :)
This makes a lot of sense. Is the follow-up to this post on the way? Have you made any concrete decisions about sup’s future?
I ask because I’ve been meaning to try sup, and possibly switch over, for a while now. This post stopped me in my tracks, though, since it implies that sup’s future probably isn’t its current incarnation as a mail client.
Switching mail clients is a big thing. It’s at least a little traumatic for most people, including me. If I’m going to do it, I’d like to know that I’m not going to get stranded and have to switch again later.
This makes a lot of sense. s the follow-up to this post on the way? Have you made any concrete decisions about sup’s future?
The followup post is on the way, and Sup’s future is pretty concrete. I went on vacation last week and spent a lot of time hacking on the foundation, so progress is progressing.
Switching mail clients is a big thing. It’s at least a little traumatic for most people, including me. If I’m going to do it, I’d like to know that I’m not going to get stranded and have to switch again later.
I understand completely. FWIW, there will be an easy upgrade path to the new Sup, and the interface will be the same, and it will be possible to get your email back out of Sup if you decide it ain’t for you.
I’m a little scared by big grand software service visions. There’s been lots of failures to make big-ass browseable/searchable stores of one person’s documents. Well at least you’re constraining yourself to textual information and it sounds like you have a specific use cases in mind.
case in point …
http://chandlerproject.org/ your post makes it sound like their server. but try not to be like them…
OK well here’s a more positive example. I’ve had a lot of success with a mac application, Journler, for taking notes that go into categories/tags/dates, plus draggable file attachments (finder replacement) plus full text search. http://journler.com/
Great plans! Can’t wait to make the switch :)
This separation between the UI, and a fast/reliable/robust component is a very good decision.
However do you think that Ruby is still the good choice for STS, I mean for the fast/reliable/robust parts?
In the mean time I think the answer is yes, but having a good high-level specification of the STS interface could help us to change it’s implementation more easily.
I’m a little scared by big grand software service visions. There’s been lots of failures to make big-ass browseable/searchable stores of one person’s documents.
I hear ya. But this isn’t really that grand of a vision, at least not incrementally from what’s there now. You take Sphinx + some kind of a document store (I’m using “files on disk” for now) + Thrift for the interface + Sup’s email threading code, and glue it together with a little logic. Hardly a Chandler.
Journler seems based on a similar idea, though I think I’m envisioning something more cohesive. Hard to say without using it. Maybe I’ll give it a try when I’m at work and have access to a macintosh apple computer.
However do you think that Ruby is still the good choice for STS, I mean for the fast/reliable/robust parts?
Definitely, because 1. I don’t want to write in any other language, and 2. the computational and IO bottleneck is the search, and that’s not in Ruby.
In the mean time I think the answer is yes, but having a good high-level specification of the STS interface could help us to change it’s implementation more easily.
Yes, definitely. I will publish the Thrift interface once it’s gelled a bit more and we call all write Erlang backends for it. :)
I like sup’s UI and focus on indexing/tagging over folders, but the idea of sup storing my mail in some opaque format is a huge turn-off. I’ve had to move my mail between clients too many times to trust it to anything but a dead-simple plaintext format like Maildir or mbox. If I wanted to be tied to a specific platform, I’d just use GMail.
I like sup’s UI and focus on indexing/tagging over folders, but the idea of sup storing my mail in some opaque format is a huge turn-off.
Sadly, that’s the only way I can reasonably support the kinds of things that I want Sup to do. Sup has never really supported mbox, Maildir, etc. very well, because it just wasn’t able to handle the concurrent modifications files in these formats undergo when there’s more than one client involved.
With the addition of non-email document types, a custom format kinda makes sense.
If it makes you feel any better, there will be tools to extract content from Sup into standard formats.
Sounds kind of like Akonadi http://pim.kde.org/akonadi/
I sincerely hope that you will rewrite sup so that tags and all the other metadata you can attach to messages will be IMAP-synchronisable. Many of us use two or more computers, and it would be useless if we had to maintain tags independently on two machines. In fact, it would send us right back into POP2/POP3 land.
I like the idea of Sup the Service, as long as there is an NCurses interface to it, as planned.
After reading about Sup on a mailing list, I considered switching fro Mutt, but decided not to due to the lack of proper mailbox handling. However, I have no objection to having messages stored in a database (perhaps I could extract all those compressed tar files of old e-mail and index the lot!). The robustness of a proper database would help – I’m thinking of atomic operations, and the data integrity guarantees provided by proper database systems.
Of course, the concept could be extended, as suggested, to blog posts, RSS feeds, NNTP newsgroups, and so forth.
On Wed, May 13, 2009 at 2:11 PM, William Morgan <comments@all-thing.net>wrote:
on quoted. settings. In “Rethinking Sup part I”: http://all-thing.net/2008/06/rethinking-sup.html, I concluded that Sup the MUA is an evolutionary dead end, and that the future lies in Sup the Service (STS). But what does that mean?
Did you ever use Zoe (back in the pre-gmail days)? It was a java app that you fed mail into and would Lucene everything and give you back a web interface with things sliced up by keywords / attachments / threads / etc.
But reworking Sup is a great time to extend its original goals. In particular, I would love for STS to handle to other types of documents besides email. I’ve always used my inbox as a mechanism for writing notes to myself. I’ve experimented briefly with reading RSS feeds through it. I’d like STS to support email, of course, but not to be limited by it.
My grand vision: STS will be a searchable, labelable, threaded view of large numbers of documents.
There’s an even more grand vision coming from developers at Google. http://wave.google.com/ http://www.waveprotocol.org/ (It’s based on XMPP, and combines features of e-mail, collaborative document editing – wiki style – and instant messages). It would be nice to bring something with these capabilities into one’s ncurses environment.
At least the specifications will be implementable in free software, and Google have foreshadowed the release of source code under free terms.
Whether this turns out to be a significant protocol on the net or not in years ahead, it’s at least worth looking at closely now.
My grand vision: STS will be a searchable, labelable, threaded view of large numbers of documents.
I read up until here and thought “cool”, but then I lost hope. I don’t think the desktop will be important in the future. Given mobile devices and ubiquitous computing, we need to focus on data synchronisation and offline operation, not the desktop.
And apart from that, I think small visions that turn into working things are preferable than grand visions.
— martin | http://madduck.net/ | http://two.sentenc.es/ “they redundantly repeated themselves over and over, incessantly without end and ad infinitum” — ibid spamtraps: madduck.bogus@madduck.net