Smoothing users’ votes

William Morgan, March 31, 2009 11:16pm
stats

In a previous post I describe how you can cook up a Bayesian framework that results in IMDB’s so-called “true Bayesian estimate”, a formula which, on its face, doesn’t look particularly Bayesian.

As my astute commenters pointed out, this formula has many simpler interpretations without needing to invoke the B word. For example, it’s a linear interpolation between two values:

θ^=λ(v)R+(1−λ(v))τ
where R is our mean vote, τ is some smoothing target, λ(v) is the smoothing weight. λ(v) can be any function, as long as it increases with v, stays between 0 and 1, and is 0 when v is 0. Those constraints give you the right behavior: with no votes, your estimate is τ exactly; as you add votes, it approaches R, and λ(v) controls how fast that happens.

This formulation naturally leads to the following question: if I’m smoothing like this to deal with paucity-of-data issues, what value of τ should I pick? IMBD uses the τ=C, the global movie mean. Intuitively that makes sense, but is it the right choice?

What’s nice about the expression for θ^ above is that the behavior we’re most interested in is when v=0, i.e. when there are no votes. In that case, θ^=τ, because of how I’ve constrained λ(v). So finding the best τ is equivalent to finding the best θ^ when v=0.

Happily, we can answer the question of the best θ^ analytically, at least if we’re happy to imagining that there is a “true” value of the movie θ.

Given θ, we can define a loss function L(θ,θ^) that describes how bad we think a particular value of θ^ is. But we don’t really know what θ is for any movie (if we did, we wouldn’t be bothering with any of this). So we can generalize that a step further and define a risk function R(θ,θ^)=Eθ[L(θ,θ^)] quantifying our expected loss: the aggregate of the loss function across all possible values of θ, weighted by the probability of each value. This gives us the tool we really need to answer the question above: the θ^ that minimizes our risk is the winner.

In the absence of any specific notions about errors, we’ll use the standard loss function for reals, squared-error loss: L(θ,θ^)=(θ−θ^)2. Then it’s just a matter of churning the crank:

R(θ,θ^)=Eθ[L(θ,θ^)]=Eθ[(θ−θ^)2]=Eθ[θ2−2θθ^+θ^2]=Eθ[θ2]−2θ^Eθ[θ]+θ^2

We can drop that first term since we’re only interested in minimimizing this as a function of τ. To find the minimum:

ddθ^−2θ^Eθ[θ]+θ^2=0−2Eθ[θ]+2θ^=0θ^=Eθ[θ]

Unsurprisingly, we see that the best estimate of θ under squared-error loss is the mean of the distribution of θ. Since we’re interested in the case where v=0, this implies that the best value to use for τ is also the mean.

So IMDB’s choice of C makes sense: the mean vote over all your movies is a great estimate of the mean of the distribution of θ.

A couple concluding points:

  1. This answer is specific to squared-error loss; if you plug in another loss function, the optimal value for τ might very well change. And you might actually have a specific model in mind for how “bad” mis-estimates are. Maybe over-estimates are worse than under-estimates, or something like that.
  2. The definition of the distribution of θ is actually completely vague above. In fact we don’t even talk about it; we just use it implicitly in our Eθ[⋅] terms. So you should feel free to plug in (the mean of) whatever distribution you believe most accurately represents your product/movie/whatever. IMDB could arguable to better by plugging in per-category means, or something even fancier.
  3. IMDB is actually a particularly bad case because movie opinions are extremely subjective. If you’re serious about modeling very subjective things, we should be talking about multinomial models, Dirichlet priors, and the like.

But the take-home message is: in the absence of a specific loss function that you really believe, smoothing towards the mean isn’t just intuitive, it’s minimizing your risk.

Discussion

To leave a new comment, enter your email address. A copy of the article will be sent to you via email.