the all-thing | 2010-07-29 19:51:24 -0400 ========================================== Bayesian hypothesis testing and decision theory ----------------------------------------------- Date: October 1, 2008 5:55pm Author: William Morgan Labels: stats URL: http://all-thing.net/old39.txt [image: evobayes.jpg] I've been doing a lot of learning at the new job. Not because people here are teaching me stuff, but more because I'm in a good position to spend a significant portion of my day learning about stuff that will help me do my job. (Which is great, and fun, and further reinforces what I know about myself by now--I'm a great self-directed learner and a very poor externally-directed learner.) One of the things I've learned is that when it comes to statistics, I'm a Bayesian. And all the crap I learned about things like hypothesis testing and maximum likelihood estimation in my stats classes now seems horribly clunky and old-fashioned to me. Let's take hypothesis testing as an example. In the classical/frequentist world, you pick an arbitrary "small enough" probability (aka 5%), find the sampling distribution of your statistic under your null hypothesis, and if it's below that threshold, say yea, else say nay. Here are some things that are wrong/bad with that approach: the 5% threshold is completely arbitrary, the sampling distribution under the alternative hypothesis is not taken into consideration (i.e. you only care about type I errors), and you don't have any way to balance the cost of type I vs type II errors. (Never mind the fact that people ALWAYS just use t-tests and ignore the fact that their datapoints are not actually distributed Normally and with the same means and variances. That, at least, I can tell you how to fix.) Compare this with the Bayesian decision theory version of hypothesis testing: you assign a cost to the two types of error, calculate the posterior probability under both conditions, based on the observations and incorporating any prior knowledge if you have it, calculate a threshold that minimizes your expected cost, and accept or reject based on that. Doesn't that just make more sense? I highly recommend the book Bayesian Computation with R [1]. (Although it doesn't actually talk about decision theory!) It has an associated blog: LearnBayes [2]. Other things to look at: William H. Jefferys's Stats 295 class materials [3] (especially these slides [4], which I'm still working my way through), and his blog for the class [5]. [1] http://bayes.bgsu.edu/bcwr/ [2] http://learnbayes.blogspot.com [3] http://quasar.as.utexas.edu/stat295.html [4] http://quasar.as.utexas.edu/courses/stat295.2007/10BayesianHypothesisTesting.pdf [5] http://bayes-rules.blogspot.com/ (Reply to this at http://all-thing.net/old39.txt.) Simpson's Paradox ----------------- Date: September 24, 2008 5:46pm Author: William Morgan Labels: stats URL: http://all-thing.net/old21.txt I found a really cool visual explanation of Simpson's Paradox on the Wikipedier. [image: 430px-Simpsons-vector.svg.png] Informally, Simpson's Paradox states that, if you and I are competing, and I do better than you in category A, and I also do better than you in category B, my overall score for both categories combined could actually be _worse_ than yours. The Wikipidia article [1] gives a real-life example: ""In both 1995 and 1996, [David] Justice had a higher batting average [...] than [Derek] Jeter; however, when the two years are combined, Jeter shows a higher batting average than Justice."" And there's also a famous legal case about Berkeley's admission rates for women from the 70's, where they were sued because the overall admission rate was lower for women than for men. Turns out that if you break it down by department, each department actually had a _higher_ admission rate for women. This all sounds crazy until you stare at the picture above for a while. The slopes of the lines are the percentages. Both solid blue vectors have smaller slopes than their corresponding solid red vectors, but when you add the them (shown as a dashed lines), the blue vectors have a bigger slope. What the picture really makes clear is that a ratio or a percentage is not a complete description of the situation. Knowing a percentage is equivalent to knowing the angle of a vector without knowing its magnitude. You can see from the picture that this isn't a weird corner case; there are _many_ choices for the second blue vector that would have the same result. It's been probably 10-12 years since I learned about Simpson's Paradox in some undergrad stats class. Now I finally really understand it. [1] http://en.wikipedia.org/wiki/Simpson%27s_paradox (One reply on this article at http://all-thing.net/old21.txt.) A brilliant idea ---------------- Date: August 24, 2008 9:13pm Author: William Morgan Labels: classification URL: http://all-thing.net/old30.txt Filter comments which contain incorrect punctuation or misspellings [1]. In this case it's YouTube comments (and apparently I'm not the only one who has a problem describing them without using the word "cesspool"), but frankly Reddit could use it just as well. The results are compelling [2], there's a very strong correlation between comment stupidity and poor spelling/punctuation. This is brilliant in its simplicity, and it would be interesting to see how this compares to an approach using StupidFilter [3], which I guess is some kind of binary SVM classifier. Advantages: 1. Far more debuggable and understandable than the output of SVM or a Bayesian classifier. 2. Much quicker to execute! 3. Wide-spread usage would have a positive effect on society as a whole. [1] http://www.chrisfinke.com/addons/youtube-comment-snob/ [2] http://www.flickr.com/photos/waxpancake/2790444683/--waddayaknow [3] stupidfilter.org (Reply to this at http://all-thing.net/old30.txt.) An awesome function ------------------- Date: August 20, 2008 12:12am Author: William Morgan Labels: proglang URL: http://all-thing.net/yesterdays_date.txt As seen in a random Reddit comment [1] def yesterdays_date(): yesterday = time.localtime() time.sleep(86400) return yesterday [1] http://www.reddit.com/comments/6wx6a/ask_reddit_whats_the_funniest_code_youve_ever_read/c053daq: (Reply to this at http://all-thing.net/yesterdays_date.txt.) Git 1.6.0 changes ----------------- Date: August 18, 2008 8:16pm Author: William Morgan Labels: git URL: http://all-thing.net/git-1.6.0-changes.txt Git 1.6.0 (just released) contains now detects Ruby class, module and method definitions in diff output. Previously it was just class names. (This patch [1] Other things I'm excited about in the new Git: 1. @git-clone --mirror@ is a handy way to set up a bare mirror repository. 2. @git-diff --check@ now checks for leftover merge conflict markers. 3. @git-stash save@ now has a --keep-index option. This lets you stash away the local changes and bring the changes staged in the index to your working tree for examination and testing. 4. @git-stash@ also has a new @branch@ subcommand to create a new branch out of stashed changes. [1] http://n2.nabble.com/-PATCH--diff:-add-ruby-funcname-pattern-td661824.html.) (Reply to this at http://all-thing.net/git-1.6.0-changes.txt.) Trollop news ------------ Date: July 30, 2008 6:58pm Author: William Morgan Labels: trollop URL: http://all-thing.net/old32.txt Looks like there was a Ruby Inside article featuring Trollop [1] a few weeks ago. Partially as a result of this, I have at least two other people contributing patches. For a project that's been around for a few years and basically had no one but me use it, that's a nice change of pace. I've also moved it over from SVN to git (hosted on Gitorious [2], which probably will help some. [1] http://www.rubyinside.com/trollop-command-line-option-parser-for-ruby-944.html [2] http://gitorious.org/projects/trollop) (Three replies on this article at http://all-thing.net/old32.txt.) Google textfile auto-titleing ----------------------------- Date: July 28, 2008 2:25pm Author: William Morgan Labels: ditz URL: http://all-thing.net/old14.txt If you search for "ditz readme" on the Googles, the correct result [1], which is a text file and not an HTML page, appears with the title "DitzのREADME". This is probably because there's a link to it titled as such in this Japanese description of a Ditz emacs mode [2]. Apparently Google prefers that title over the link just called "README" on the the Ditz main page. [1] http://ditz.rubyforge.org/README.txt [2] http://d.hatena.ne.jp/antipop/20080412/1208010913 (Reply to this at http://all-thing.net/old14.txt.) Ditz 0.4, and the magic of Ruby DSLs ------------------------------------ Date: July 27, 2008 6:21pm Author: William Morgan Labels: ditz URL: http://all-thing.net/old26.txt I've just released Ditz 0.4. The big-ticket item in this release is the plugin system, which makes it very easy to tweak Ditz's models, views and controllers. There's an included git plugin which does some nice things like linking git commits and git branches to individual Ditz issues. The new bash completion is pretty nice too. The completion code has been reworked a bit and now ties in very nicely with the argument processing. Check out this code from the Ditz's controller (operator.rb [1], for those following along from your repo): operation :start, "Start work on an issue", :unstarted_issue def start project, config, issue ## ... end Just by calling the operation method, we get: * Argument checking. There must be one argument to 'ditz start', and it must be an unstarted issue. Any violations are handled nicely for us without having to invoke the method. * Help messages in 'ditz help' and 'ditz help start'. * Argument completion. Running 'ditz start ' outputs a list of possible completions for a command (in this case, all unstarted issues) and then exits. Shell completion scripts can parse this output and present it to you when you hit tab. So that's a little DSL that I think turned out well. Writing a plugin is also nicely DSLified. Here are some examples from plugin/git.rb: class Issue field :git_branch, :ask => false def git_commits ## ... end end Here we reopen the Issue class and add a field called @git_branch@, and we specify that the UI shouldn't ask for this field when an issue is created, since I decided that would be too annoying. (We'll see how we allow the user to explicitly set it below.) We also add a method that's responsible for actually getting the commits out of git. Since our configuration file is just a Ditz model object, we can do the same thing to add the configuration parameters we need: class Config field :git_commit_url_prefix, :prompt =>"URL prefix (if any) to link git commits to" field :git_branch_url_prefix, :prompt => "URL prefix (if any) to link git branches to" end We'll use those two fields to add some links when we generate HTML. Adding commands to Ditz's controller is just as easy. Just reopen the class: class Operator operation :set_branch, "Set the git feature branch of an issue", :issue, :maybe_string def set_branch project, config, issue, maybe_string ## ... end end So now we have a @set-branch@ command that takes an issue name, and an optional branch name. And it's a first-class citizen alongside every other command: shows up in the help page, has argument auto-completion, etc. Finally, let's see how we modify the views. One thing we'd like to see is the git branch for an issue, if it's been set. class ScreenView add_to_view :issue_summary do |issue, config| " Git branch: #{issue.git_branch || 'none'}\n" end end Here we've opened up the ScreenView class (which is used for generating the screen output, as opposed to the HTML output) and added a closure to the summary section, which prints out the value of the model field we added above. The HTML version is similar: class HtmlView add_to_view :issue_summary do |issue, config| next unless issue.git_branch [{ :issue => issue, :url_prefix => config.git_branch_url_prefix }, < Git branch: <%= url_prefix ? link_to([url_prefix, issue.git_branch].join, issue.git_branch) : h(issue.git_branch) %> EOS end end The HTML generation returns ERB, and a hash of variables necessary for resolving it. (In this case we could also have used string substitution, but that's not always the case--you might want to make use of variables that are only available at generation time.) Note that it also makes use of some of the convenient helper functions (like @link_to@ and @h@), which I've helpfully defined for you in html.rb. So that's how to modify ditz's models, views and controllers in one easy go. You can add fields and helper methods to model objects (including the configuration object), you can add commands to the controller, and you can add view elements to the screen and HTML output. For reference, the complete source code to the git plugin is here [2]. [1] http://gitorious.org/projects/ditz/repos/mainline/blobs/master/lib%2Foperator.rb [2] http://gitorious.org/projects/ditz/repos/mainline/blobs/master/lib%2Fplugins%2Fgit.rb%22 (Reply to this at http://all-thing.net/old26.txt.) Vim ruby syntax comment reformatting ------------------------------------ Date: July 20, 2008 2:44pm Author: William Morgan Labels: ruby, vim URL: http://all-thing.net/old22.txt The vim ruby syntax seems to screw up comments that have multiple hashes. E.g. I like to differentiate ### section heading comments, ## non-inline comments, and x = a + b # inline comments But reformatting the comments (e.g. with "gq}") always screws them up, unless you do: $ mkdir -p ~/.vim/after/syntax $ cat > ~/.vim/after/syntax/ruby.vim set comments=n:# which tells vim that multiple hash marks are ok. (Four replies on this article at http://all-thing.net/old22.txt.) Rethinking Sup part II ---------------------- Date: July 20, 2008 1:58pm Author: William Morgan Labels: sup URL: http://all-thing.net/old13.txt In Rethinking Sup part I [1], I concluded that Sup the MUA is an evolutionary dead end, and that the future lies in Sup the Service (STS). But what does that mean? One thing I want to make clear it does _not_ mean is any abandonment of the Sup curses UI. That particular "user experience" has been refined over the past few years to become my ideal email interface. It would be silly to throw that away. What _will_ happen to the curses code is that it will become one client among (hopefully) many. Once there's a clear delineation between UI and backend, you can make a UI choice independent of making a choice to use Sup in the first place. You can run sup-curses-client if you want. Or you can build a web interface, or an Openmoko interface. Working with ncurses has always been the least enjoyable part of Sup, so maybe I'll actually enjoy learning Javascript. What backend functionality will STS actually provide? If I were simply reworking Sup into a client and a server, the obvious answer would be "a searchable, labelable, threaded view of large amounts of email". But reworking Sup is a great time to extend its original goals. In particular, I would love for STS to handle to other types of documents besides email. I've always used my inbox as a mechanism for writing notes to myself. I've experimented briefly with reading RSS feeds through it. I'd like STS to support email, of course, but not to be limited by it. My grand vision: _STS will be a searchable, labelable, threaded view of large numbers of documents._ You can throw whatever you want in there, and STS will store it, thread it, and let you label and search for it. Email, RSS feeds, notes, jabber and IRC logs, web pages, RI documents—I want you to be able to throw them all in there. I want you to be able to annotate any of those things by adding notes and threading them against the original objects. Basically I want STS to be the primary tool you use for organizing and recalling all the textual information you've ever encountered in your life. Cool, huh? There's another convenient benefit to this transformation: no one will expect STS to act like a MUA. STS does its own storage. You add your email and your other documents to the server and then you can throw those files away (or not). There are no more questions of supporting IMAP or various mbox dialects or "why doesn't Sup treat Maildir correctly". The files are in STS, and once they're their, they're out of your hands. You'll be able to export them, of course, and if you're crazy you might be able to write an IMAP _server_ translation layer for STS, but there will be no more expectation of realtime Maildir handling. As I explained in part I, that's a game I don't want to play. STS is a grander vision than a MUA, and it no longer has to be hobbled by the constraints of being expected to act like one. Some other nice benefits of reworking Sup into SYS: * You'll be able to run multiple clients at once. * It's an opportunity to rework some things. For example, one of the most noticeably slow operations in Sup ("Classic") is assembling a large thread. This is because I made a decision early on to do all threading at search time. That made certain things easier (in particular, I could change the threading model without having to rescan the entire index), but in retrospect the cost is too high. STS will maintain document trees directly. * I can replace Ferret with Sphinx. It's been a good couple years, but the periodic non-deterministic index corruption that's been an issue for over a year [2] is an exit sign to me. Working with Sphinx is nowhere nearly as nice as working with Ferret, but speed and stability go a long way. I've been working on the code for STS on and off for the past couple weeks and it's slowly starting to come together. Once the major components have at least been all sketched, I will host a git repo. [1] http://all-thing.net/2008/06/rethinking-sup.html [2] http://ferret.davebalmain.com/trac/ticket/279 (19 replies on this article at http://all-thing.net/old13.txt.) Pages ----- * Page 1: http://all-thing.net/index.txt * Page 2: http://all-thing.net/index/1.txt * Page 3: http://all-thing.net/index/2.txt * Page 4: http://all-thing.net/index/3.txt * Page 5: http://all-thing.net/index/4.txt * Page 6: You're reading it. * Page 7: http://all-thing.net/index/6.txt * Page 8: http://all-thing.net/index/7.txt This delicious text version served up by Whisper .