The scripts used for the empirical evaluations in the NUTS paper are not included with the NUTS software, so in some respects I’m not certain what was done, but the description in the paper is fairly complete.

The foundation for any empirical assessment of MCMC methods is some way of estimating how good the MCMC estimates are, which could be the autocorrelation time, or the variance of the estimate, or the effective sample size (ESS, the equivalent number of independent points). The NUTS paper uses autocorrelation time, from which they compute the effective sample size.  They estimate autocorrelations using means and variances found from a very long run, not from the run being evaluated, which avoids one possible source of drastically wrong results in such evaluations. (For the zero-mean multivariate normal example I discuss below, they presumably plugged in the true mean of zero and the true variances, rather than use a long run.) As their measure of overall performance for a sampler, they used its worst performance (in terms of autocorrelation time or ESS) over all variables (looking at both mean and variance estimates for these variables). Focusing on the worst estimate seems reasonable in a way, but looking at only this one number could be misleading if the estimate that is worst is different for different samplers.

One aspect of their evaluation (as described in the paper) is incorrect. In the Appendix, they say that they estimate autocorrelation time by summing autocorrelation estimates at all lags up to the first lag at which the autocorrelation estimate is less than 0.05. Since the true autocorrelation can sometimes be negative, this is not valid, in general — it would be for some simple random-walk Metropolis methods, for which autocorrelations are all positive, but not for HMC, for which negative autocorrelations are quite possible. However, for reversible methods (which includes HMC), the sums of consecutive pairs of autocorrelations are always positive. This provides the basis for a similar approach that is valid. See here for a review and assessment of this and other approaches to estimating autocorrelation time.

Assuming that this is an error in the actual code (not just the paper), it could affect the empirical results quite a bit, but I’ll have to ignore this possibility in the remaining discussion.

The first distribution that Hoffman and Gelman test NUTS on is a 250-dimensional multivariate normal distribution with mean zero and a covariance matrix randomly drawn from an inverse Wishart distribution. They claim that NUTS performs better on this distribution than HMC (by a factor of about three) even when HMC is optimally tuned. This seems strange, since NUTS is just an automated way of tuning HMC. As I’ll show below, this claim is indeed likely to be due to deficiencies in the experimental methodology.

Due to rotational invariance, performance of HMC on a multivariate normal is determined by the set of eigenvalues of the covariance matrix, with the inherent difficulty of the problem being proportional to the square root of the ratio of the largest eigenvalue to the smallest eigenvalue. This is the ratio of the standard deviation in the least confined direction to the standard deviation in the most confined direction, and should be roughly the number of leapfrog steps in a trajectory of optimal length, using the optimal stepsize. (Performance by Hoffman and Gelman’s metric isn’t quite rotationally symmetric, however, since they look only at autocorrelation times for the 250 variables, not all linear combinations of them.) The evaluation of HMC and NUTS on this distribution would be much more informative if the eigenvalues of the covariance matrix were reported, and if the number of leapfrog steps in HMC trajectories (not just the product of stepsize and number of steps) were reported.

I’ll return below to the multivariate normal example, but first I’ll mention the other distributions on which HMC and NUTS are evaluated in the paper. Two of these other distributions are posterior distributions for logistic regression models. The results on these distributions are more what one might expect — NUTS performs about as well as HMC with optimal tuning. One slightly strange phenomenon is visible in the results presented in Figure 6, however. For both logistic regression problems, the performance of HMC drops drastically as the trajectory length is increased from the optimal value to a value larger by a factor of 1.5. I would expect a more gradual decline in efficiency.  It would be nice to know why this occurs.

The last distribution they evaluate NUTS and HMC on is the posterior distribution for  a stochastic volatility model.  Here again, they claim that NUTS performs better than HMC, even when the latter is optimally tuned. I’m not sure what is going on here, but one thing I notice in the lower left plot in Figure 6 is that the performance of NUTS seems to be bimodal — some of the runs are assessed as being three times better than the best HMC runs, but others are assessed as being worse. All I can think of to explain this bimodality is that the adaptive stepsize selection might be reaching either of two stable values. The advantage of some NUTS runs over the best HMC runs might be explained by the same factors that lead to this result for the multivariate normal.

Since the actual multivariate normal distribution used in the paper isn’t reported, I’ll look at performance of NUTS and HMC on another multivariate normal, the same 30-dimensional multivariate normal distribution that I used in my previous post, in which the 30 components are independent, with standard deviations of 110, 100, 1.1, 1.0, and 26 values equally spaced between 8 and 16. This may in any case be a better demonstration example than one in which the standard deviations are chosen randomly.

I used the NUTS software for these experiments, with automatic stepsize adaptation for both HMC and NUTS (the adaptation always converged to a reasonable value of around 1.63 for HMC and around 1.73 for NUTS). I did 10,000 burn-in iterations, followed by 250,000 sampling iterations, which I divided into 400 batches of size 625. I estimated the mean of each variable and the second moment of each variable (equal to the variance, since the true mean is zero) for each of the 400 batches, and looked at the variance of these batch estimates as a measure of the performance of the sampler, not accounting for computation time yet. I then multiplied the variances of these batch estimates by the number of gradient evaluations for the whole run to get figures of merit that account for computation time. I used this Matlab script for these experiments (and then moved the results to R for plotting).

Here are the results for estimating the means of each of the 30 variables (numbered 1 to 30 by decreasing standard deviation), for HMC with trajectory lengths of 100, 170, and 200 leapfrog steps (and hence trajectory lengths of about 163, 277, and 326 time units since the stepsize is about 1.63):

The left plot gives the variances of the estimates, not accounting for computation time; the right plot adjusts for computation time (gradient evaluations). Smaller is better in both plots. Note that the vertical scale is logarithmic, with horizontal lines at each power of ten.

These results look rather peculiar! Notice how the variances of the mean estimates for variables 3 to 28, with standard deviations declining from 16 to 8, don’t decline as one might expect, but rather go up and down, with peaks at positions that change with the trajectory length.

This behaviour is due to a phenomenon discussed in Section 3.2 of my review of Hamiltonian Monte Carlo. Hamiltonian dynamics for this simple distribution without dependencies between variables will be periodic for each variable. If the period for a variable happens to match (or be near) the trajectory length, the end-point of a trajectory will be close to the start point, with the result that sampling for that variable is very poor. (A related problem  happens when the trajectory length is half the period.) Note that this problem can arise for any multivariate normal distribution, with the poorly-explored directions being the eigenvectors of the covariance matrix, not necessarily the individual variables.

The cure for this is to randomly vary the trajectory length (randomly varying the stepsize, or the number of leapfrog steps, or both) over some moderate range. I modified the hmc_da function in the NUTS software to do this, changing the computation of the number of leapfrog steps from

L = max(1, round(lambda / epsilon));

to

L = max(1, round((0.9 + rand/5) * lambda / epsilon));

which has the effect of multiplying the number of leapfrog steps by a random factor uniformly distributed between 0.9 and 1.1.

Here are the results after this modification:

The variances of the mean estimates still go up and down a bit (maybe a wider random range would be better), but much less than before.

So we see that one can get a misleadingly-bad impression of the performance of HMC on a multivariate normal distribution from experiments in which the trajectory length is not varied randomly . On less simple distributions, the exact periodicities that happen with a multivariate normal may not be present, but there can be approximate periodicities, which might still happen to nearly match a fixed trajectory length, with a consequent bad effect on performance. This is why randomly varying the trajectory length is a recommended part of standard HMC methodology. (However, this is unnecessary with “windowed” HMC, as long as the window size is a non-negligible fraction of the trajectory length.)

So, how does NUTS perform on this distribution? Here are the results, together, for comparison, with the variances of mean estimates from an equal-size sample of points drawn independently (omitted for the computational cost plot, where it would not be meaningful):

For the first two variables, with the largest standard deviations, the variances of the NUTS estimates of the mean are about 30 times bigger than the variances when the sampled points are independent.  Compared to HMC with random trajectory length with median 200, these NUTS estimates have a variance that is about 25 times bigger. After adjusting for computational cost, the NUTS estimates of the means of these two variables are 7 times less efficient than the HMC estimates.

NUTS looks much better for the other variables, with smaller standard deviations. Indeed, the NUTS estimates are often better than those obtained with independent sampling! This is possible because NUTS (and HMC) can produce negative autocorrelations. However, for negative correlations to show up with these variables, NUTS must often be using trajectories that are much shorter than are needed for efficient exploration of the large-standard-deviation variables, which of course is consistent with it estimating the means of those variables inefficiently.

Here are plots showing the performance of HMC with random trajectory length and of NUTS when estimating the variances (rather than the means) of the 30 variables:

HMC is about a factor of two more efficient than NUTS at estimating the variances of the two variables with the largest standard deviations, but is less efficient at estimating the variances of the other variables.

One might think that these results show that there’s a tradeoff — HMC better for variables with large standard deviation, NUTS better for variables with small standard deviation. But I think this isn’t really so. First, it is often the absolute magnitude of the variance of estimates that is important, and in that respect HMC is clearly better. (Consider, for instance, estimating regression coefficients for covariates that have been standardized.) When that’s not so — when you need precise estimates of the means of variables that have small standard deviations — it’s easy to alternate HMC updates designed to sample well for the variables with large standard deviations with other, much cheaper, updates designed to sample well for the variables with small standard deviations (eg, univariate slice sampling updates, or just HMC with a smaller trajectory length). Using the windowed HMC method would also help, since rejected transitions in windowed HMC will usually move a small distance, helping with the sampling for variables with small standard deviations. (Note: as mentioned before, due to the rotational invariance of HMC, all this discussion applies also when the “variables” are actually various directions in the space, that aren’t necessarily the coordinate axes being used.)

One conclusion I draw from this evaluation is that the better performance of NUTS over HMC (optimally tuned) shown in the evaluations in the NUTS paper is likely an artifact of not randomizing the HMC trajectory length, and of evaluating performance by the worst effective sample size for any variable (not accounting for some variables being easy to handle by other methods). Another conclusion is that the premature U-Turn problem that I discussed in my first post on NUTS can be a problem in practice, since otherwise NUTS would estimate the means and variances of the large-standard-deviation variables in this example better. As I discussed there, I think there is scope for improving the way that NUTS decides when to stop a trajectory, and thereby make it a more generally-useful sampling method.

It’s a good paper, which addresses two big barriers to wider use of Hamiltonian Monte Carlo — the difficulties of tuning the trajectory length and tuning the stepsize to use when simulating a trajectory. The name “No-U-Turn Sampler” (NUTS) comes from their way of addressing the problem of tuning the trajectory length — repeatedly double the length of the current trajectory, until (simplifying a bit) there is a part of the trajectory that makes a “U-Turn”, heading back towards its starting point. This doubling method is clever, and (as I discuss below) one aspect of it seems useful even apart from any attempt to adaptively set the trajectory length. They also introduce a method of adapting the stepsize during the burn-in period, so as to achieve some desired acceptance probability.

However, I don’t think these are completely satisfactory ways of setting trajectory lengths and stepsizes. As I’ll discuss below, these problems are more complicated than they might at first appear.

The biggest advantage of HMC over simple random-walk Metropolis or Gibbs sampling methods is that it can propose to move to a distant point, with a high probability of acceptance. This gets rid of the inefficiency of a random walk, in which taking n small steps in random directions is likely to move you only about √n steps away from where you started.  HMC can propose the end-point of a trajectory found using n steps that move more-or-less consistently in one direction, ending up about n steps from the starting point.  To get the full advantage of this, the trajectory has to be long enough (but not much longer) that in the least constrained direction the end-point of the trajectory is distant from the start point. This could require that a trajectory be much longer than the length at which it starts to move back towards the starting point, which (roughly speaking) is the point where the No-U-Turn method would stop.

Such premature U-Turns can occur when some directions are highly constrained, limiting how big the stepsize can be, while some other directions are much less constrained, and hence will take many steps to move across, and yet other directions are constrained to an intermediate degree. In  directions with intermediate constraints, the trajectory can reverse direction long before the least constrained direction has been explored. This can produce a “U-Turn” when the trajectory is much shorter than is optimal.

Consider an example of sampling from a 30-dimensional multivariate normal distribution in which the 30 components are independent, with standard deviations of 110, 100, 1.1, 1.0, and 26 values equally spaced between 8 and 16. (Since the operation of HMC is invariant to rotation, behaviour on this distribution is the same as on any 30-dimensional multivariate normal for which the square roots of the eigenvalues of the covariance matrix have these values.) The smallest standard deviation of 1 limits the stepsize to less than 2 (at which point the dynamics becomes unstable). A stepsize of 1.5 gives reasonably good results. Exploring the less constrained directions where the standard deviations are 110 and 100 will then require hundreds of steps.

I simulated six trajectories from starting points randomly drawn from this distribution.  Each trajectory was for 300 leapfrog steps with a stepsize of 1.5, a duration of 450 time units.  Below is a plot (produced using this R script, which calls this HMC routine) of the distance from the start point for the points after each leapfrog step in these trajectories (trajectories that were rejected are plotted as dotted lines).

As one can see, the distance from the start point often does not increase monotonically for the time needed to reach a distant point.  It’s hard from this plot to tell exactly what will happen when using the  No U-Turn Sampler, since it looks for U-Turns in subtrees, which don’t always have the same start point as the entire trajectory. It’s clear, though, that it will be quite possible for NUTS to stop doubling prematurely, before reaching a trajectory length that will produce good movement in the least-constrained direction. However, we also see that occasionally the distance from the starting point does increase monotonically for a suitable time, so perhaps NUTS will at least occasionally use a suitable trajectory length.

I think there is scope for improving NUTS in this respect, by finding some better criterion for stopping the doubling process. One simple modification that might be desirable is to simply refuse to stop doubling until some minimum trajectory length is reached — a minimum length of eight might be about right. This will be less efficient for problems in which the optimal trajectory length is smaller than this, but such problems are easy anyway, and will still be easy even if done up to eight times less efficiently.

Once it has stopped doubling, NUTS has an interesting way of sampling a point from the final trajectory.  Rather than just picking a point randomly from amongst those that are eligible, it first tries to move to the half of the trajectory that doesn’t contain the starting point, then if that is rejected, to move to the quarter of the trajectory that doesn’t contain the starting point but is in the same half as the starting point, and so forth, attempting to move away from the starting point by successively smaller amounts. This should produce greater movement on average than randomly selecting a point that might, just by chance, turn out to be close to the starting point.

This technique ought to also be applicable to my “windowed HMC” method (see Section 5.4 of my review), most obviously when the window size is half the trajectory length. It seems that it should be easy to try this out by just modifying the NUTS program to do a fixed number of doublings, ignoring whether the trajectories are doing U-Turns or not. The discussion in the NUTS paper mentions that some of the advantage seen for NUTS over standard HMC might be due to it having some characteristics of windowed HMC. Comparing adaptive NUTS with NUTS with a fixed number of doublings (essentially an improved version of windowed HMC) would separate these effects.

The other big issue when tuning HMC is how to choose a stepsize for the leapfrog updates that will be small enough to give a reasonably high probability of accepting trajectories, but not unnecessarily small, which would waste computation time. NUTS addresses this issue by adapting the stepsize during a burn-in period, so as to achieve a specified acceptance probability. For example, if one assumes that the distribution being sampled can be viewed as consisting approximately of many replications of some distribution, the optimal acceptance rate is 0.65 (see Section 4.4 of my review), so this might be the target of the adaptation.

I expect that this will work well for many problems. However, in some problems it may fail disastrously. This could happen if the stepsize that produces an average acceptance probability of 0.65 is much smaller in some regions of the distribution than in others. If the sampler is started in the region that allows a large stepsize, the adaptation may converge on that large stepsize. The sampler will then very likely never visit the other region, where the stepsize needs to be much smaller, and the answers obtained will be drastically wrong.

You might think that instead the sampler would at some iteration move to the region that needs a small stepsize, and then get stuck there, as all the proposals are rejected — a problem that is easily diagnosed. Unfortunately, that’s not what will happen. Since the sampler leaves the correct distribution invariant, if it would be stuck for a very long time in some region, it must also be very unlikely to enter that region. This is a potential problem for any global Metropolis algorithm, for which acceptance probabilities can be very low if a bad proposal distribution is chosen, especially in high dimensions, but is particularly severe for HMC, since a stepsize that is beyond the stability limit for the dynamics will produce exponential growth in the error, and an extremely small acceptance probability. See Section 4.2 of my review for further discussion.

One way of alleviating this problem is to choose the stepsize to use for a trajectory randomly, from some moderately broad distribution. One could still adaptively choose the mean of this distribution. Occasional choices of a small stepsize would allow the sampler to enter regions that require such a small stepsize, and either force the mean of the stepsize distribution down, or at least allow the problem to be diagnosed from the presence of some long runs of rejected trajectories.

I will discuss another reason to randomly choose the stepsize or number of leapfrog steps in a second post on NUTS (coming, I hope, in a few days), in which I will discuss the empirical evaluations done in the NUTS paper. This post will also shed some light on whether the possibility of a premature U-Turn that I discuss above is likely to be a problem in practice.

This software differs from other more-or-less general MCMC packages in several respects, all but one of which make it, I think, a much better tool for serious MCMC applications. Here are some highlights:

• The distribution to be sampled is defined by an R function.  This allows for full flexibility in defining models. For simple models, it is a bit clumsier than a BUGS specification, but one could address this by “compiling” a BUGS-like specification into such an R function (though I have no plans to write such a compiler myself).

• The Markov chain updates are also defined by R functions, which can easily be written by the user. Of course, some updates, such as random-walk Metropolis, are pre-defined.

• The interface between the R functions defining distributions and the R functions defining Markov chain updates converts between their different preferred representations of the state.  A general-purpose update function (eg, Metropolis) is most naturally written to operate on a state that is a simple vector. But a function defining the posterior distribution for a Bayesian model is most naturally written to work with a state that is a list of vectors, with named elements for various types of parameters, hyperparameters, etc.

• The top-level function for running an MCMC simulation allows one to combine several different types of updates, which may each operate on only part of the state. This is often necessary for efficient MCMC sampling for a complex model.

• GRIMS supports incremental computation, in which the log probability density for a state is quickly recomputed after only a subset of the state variables change. This is a crucial capability for some sampling methods, such as Metropolis updates with proposals that change a single variable in the state. This support depends, of course, on the function that computes the log density being written to cache suitable intermediate quantities, and make use of them when possible.

• Updates that use the gradient of the log density are supported, provided the function defining the distribution is able to compute the gradient.

• The state can be supplemented with auxiliary “momentum” variables, allowing full use of sampling methods based on Hamiltonian dynamics.

• GRIMS has flexible facilities for specifying tuning parameters of update methods (eg, proposal standard deviations), and for collecting information (eg, acceptance rates) about how the update methods are operating.

There are other additional aspects of the design (or extensions to it that I envision) that are intended to support both complex applications and complex sampling methods, including methods like tempered transtions and ensemble MCMC in which a whole series of Markov chain updates are nested within a complex outer update (though nothing that elaborate is implemented yet). It’s not completely general, however. For example, with it’s present design, GRIMS can’t easily handle states of varying dimensionality.

So what’s the one undesirable respect in which GRIMS differs from other MCMC packages? Speed. Though I haven’t quantified this yet, GRIMS is likely to be rather slow, due to the overhead of implementing the nice things listed above in R. This is one of my motivations for speeding up R, though ironically, GRIMS has ended up making heavy use of some R operations, such as subscripting lists with strings using “[[...]]”, that I haven’t (yet) looked at speeding up.

We’ll have to see how fast or slow it ends up. In any case, speed isn’t essential to all uses of GRIMS. Since much of my research is on new MCMC methods, I want a better environment for quickly trying out new MCMC methods on a variety of applications. I also want these new MCMC methods, as well as new Bayesian models that use MCMC, to have implementations that are easily accessible to statisticians, for which an R function is a lot better than a C program.

If any readers want to try out this rather unpolished preliminary version of GRIMS (or just read the documentation), I’d be interested in your comments.

UPDATE: I’ve put up a new version that fixes some bugs, adds a Gibbs sampling / overrelaxation update for normals, and adds some tests. There may still be plenty of bugs remaining.

Greenpeace recently released a report that said if the Internet were a country, it would be the fifth-largest consumer of energy, largely because of the massive data centres that run unseen in the background. The group estimated that the centres will use 1.9 billion kilowatt hours of electricity by 2020 — more than the amount currently used by Canada, France, Germany and Brazil combined. (The average US home uses 8,000 kilowatt hours a year.)

An exercise for the reader: How many logical fallacies, arithmetic errors, or contradictions of common knowledge can you find in this passage?

I haven’t tried to determine whether these fallacies originate in the (unidentified) Greenpeace report, or are original to the Globe and Mail.

Introduction to Probability with R, Kenneth Baclawski, Chapman & Hall / CRC.

Probability with R: An Introduction with Computer Science Applications, Jane M. Horgan, Wiley.

I’ve now had a look at both of these textbooks.  Unfortunately, they are both seriously flawed.  Even more unfortunately, although some of the flaws in these books are particularly striking, I’ve seen similar, if usually less serious, problems in many other textbooks.

Introduction to Probability with R seemed from the title and blurb to be quite promising.  I began looking at it by just flipping to a few random pages to see what it read like.  That wasn’t supposed to be a serious evaluation yet, but in only a couple minutes, I’d found two passages that pretty much eliminated it from further consideration.  Here is the introduction to parametric families of distributions on pages 56 and 57:

… The distributions within a family are distinguished from one another by “parameters”… Because of the dependence on parameters, a family of distributions is also called a random or stochastic function… Be careful not to think of a random function as a “randomly chosen function” any more than a random variable is a “randomly chosen variable.”

Now, I’ve never before seen a parametric family of distributions called a “random function” or “stochastic function”.  And I’ve quite frequently seen “random function” used to mean exactly a “randomly chosen function”.  Where the author got his terminology, I’ve no idea.  For good measure, this same passage refers to the p parameter of a binomial distribution as the “bias”, and has a totally pointless illustration of a bunch of gears grinding a value for n and a “bias” into a distribution.

So, maybe he uses non-standard notation, but could the content be good?  Here’s what’s on page 131:

Main Rule of Statistics.  In any statistical measurement we may assume that the individual measurements are distributed according to the normal distribution, N(m,σ²).

To use this rule, we first find the mean m and variance σ² from information given in our problem or by using the sample mean… and/or sample variance… defined below. We then compute using either the pnorm or the qnorm function.

As stated, the main rule says only that our results will be “reasonable” if we assume that the measurements are normally distributed. We can actually assert more. In the absence of better information, we must assume that a measurement is normally distributed. In other words, if several models are possible, we must use the normal model unless there is a significant reason for rejecting it.

(The author continues his avoidance of standard terminology in the passage above by denoting the sample mean of x₁, …, x_n by m with a bar over it and the sample variance by σ² with a bar over it, which I haven’t tried to reproduce.)

Sometimes, an occasional incorrect passage in a textbook does no harm, if corrected, and can even make for an interesting example in lecture, but when a textbook emphatically states completely erroneous nonsense it would be a disservice to students to force them to buy it.  The passage above reads like a parody of a statistics textbook — too many of which say how important it is to think carefully about what model is appropriate for your problem, but then proceed to use a normal distribution in all the examples without any discussion.  Still, lip-service to good practice is better than nothing, and much better than insistent advocacy of bad practice.

Probability with R: An Introduction with Computer Science Applications seemed from its title and blurb to be even closer to what I need for my course.  For this book, two minutes of flipping through it did not provide sufficient grounds for rejection, so I started looking at it more closely, including reading it systematically from the beginning.  I found lots of careless errors (some corrected in the on-line errata), and lots to quibble about, along with nothing that was particularly impressive, but it wasn’t until page 83 that I encountered something seriously wrong:

Returning to the birthday problem …, instead of using permutations and counting, we could view it as a series of k events and apply the multiplication law of probability.

B_i is the birthday of the ith student.

E is the event that all students have different birthdays.

For example with two students, the probability of different birthdays is that the second student has a birthday different from that of the first,

that is, the second student can have a birthday on any of the days of the year except the birthday of the first student.

With three students, the probability that the third is different from the previous two is

that is, the third student can have a birthday on any of the days of the year, except the two of the previous two students.

The example continues like this, with equations that on the right side have numerical values that are correct (in terms of the subsequent explanation in words), while on the left side are probability statements that are complete nonsense — since of course B_i, “the birthday of the ith student” is not an event. Nor can I see any alternative definition of B_i that would lead to these probability statements making sense.

Maybe “anyone” could make a mistake like this, but maybe not  — I do wonder whether the author actually understands elementary probability theory.  I lost all confidence in her ability to apply it in practice on reading the following on page 192, in the section on ”Machine learning and the binomial distribution”:

Suppose there are three classifiers used to classify a new example. The probability that any of these classifiers correctly classifies the new case is 0.7, and therefore 0.3 of making an error. If a majority decision is made, what is the probability that the new case will be correctly classified?

Let X be the number of correct classifications made by the three classifiers. For a majority vote we need X≥2. Because the classifiers are independent, X follows a binomial distribution with parameters n=3 and p=0.7…

We have improved the probability of a correct classification from 0.7 with one classifier to 0.784 with three…

Obviously, by increasing the number of classifiers, we can improve classification accuracy further…

With 21 classifiers let us calculate the probability that a majority decision will be in error for various values of p, the probability that any one classifier will be in error…

Thus the key to successful ensemble methods is to construct individual classifiers with error rates below 0.5.

No, I haven’t omitted any phrase like “assuming the classifiers make independent errors”.  The author just says “because the classifiers are independent” as if this was totally obvious.  Of course, in any real ensemble of classifiers, the mistakes they make are not independent, and it is not enough to just produce lots of classifiers with error rates below 0.5.

Many, many textbooks give examples in which they say things like “assume that whether one patient dies after surgery is independent of whether another patient dies” without considering the many reasons why this might not be so.  But at least they do say they are making an assumption, and at least it is possible to imagine situations in which the assumption is approximately correct.  There is no realistic machine learning situation in which multiple classifiers in an ensemble will make errors independently, even approximately. The example in the book is totally misleading.

As well as being seriously flawed, neither of these books makes particularly good use of R to explain probability concepts or to demonstrate the use of simulation. The fragments of R code they contain are very short, basically using it as a calculator and plot program.

So, that’s it for these books.  If any readers know of good books on probability that either have good computer science applications, or use R for simulations and to clarify probability concepts, or preferably both, please let me know!

Assessing how much these patches speed up R is difficult. First of all, the speedup varies tremendously with the type of program. It also varies quite a bit with the machine and compiler used to run R. Finally, it varies in apparently random ways — changing code in one part of the R interpreter can change the speed of operations that never use the modified code by plus or minus 5% or more.  This is presumably due to the change altering the exact addresses of other code segments, with consequent effects on alignment of memory fetches or on cache behaviour.

Nevertheless, here is a comparison of R 2.13.0 without modification and with all my patches applied, with and without compilation of R functions. The tests were done with an Intel X5680 processor running at 3.33GHz in 64-bit mode using gcc 4.4.4 under Red Hat Linux with default R configuration parameters. The tests use my suite of speed tests for R.

Here are some highlights:

• For programs dominated by general interpretive overhead, such as the first test listed (“em”), there is a speedup by a factor of up to about 1.5.  If these programs are compiled with the R bytecode compiler, the speedup is less, but not negligible (up to a factor of about 1.1).

• Vector and matrix subscripting is substantially faster, for both interpreted and compiled programs.  The speedup is mostly when extracting subsets (for instance, r <- v[100:200]), with only a small improvement when replacing a subset (for instance, v[100:200]<-r). Here are some examples:
  • Operations such as A[100:200,300:400] are sped up by a factor of about 3.4.
  • Operations such as A[100:200,10] are sped up by a factor of about 1.8.
  • Operations such as A[,10:11], with A having 300 rows, are sped up by a factor of about 4.3.
  • Accessing a single element of a logical or complex vector is sped up by a factor of about 1.7.
  • Accessing a fairly big range of elements in a vector (eg, a[1:100]) is sped up by a factor of about 1.6.
  • Forming a vector with one element deleted (for example, a[-100], with a being of length 1000) is sped up by a factor of about 5.5.

• The interpreter now sometimes avoids creating a sequence of integers when it is just used as a subscript or in a for statement. This is partly responsible for the speedups detailed above for subscripting operations such as a[1:100].  A for statement such as for (i in 1:1000000) will now not actually create a vector of length 1000000, but just behave as if it had.

• Vector dot products (done with %*%) are sped up by a factor of 10 for long vectors.  Matrix times vector and vector times matrix products with the vector being of length 1000 are sped up factors ranging from 3.2 to 5.5.  General matrix-matrix products are sped up substantially as long as at least one dimension of the result matrix is not large.  For discussion of the issues involved here, see my previous post.

• Accessing an element of a list by name with the $ operator is sped up by a factor of about 1.5.

• Argument matching when a function is called has been sped up. The speedup for this operation alone isn’t too clear from the tests, but is probably a factor of 1.3 or more. Somewhat as a by-product of this patch, some primitive operations such as rep have also been sped up (they had used an inefficient kludge for their argument matching). For instance, rep(1,length=100) is sped up by a factor of 1.5.

I’ll now switch to working on other projects for a while, but I plan to get back to R work in August.  In the meantime, I will be posting a bit more on what some of these patches are doing.  I’m also interested, of course, in hearing of any problems that people may have installing the patches, in any bugs that they find, and in reports of how much effect the patches have on the speed of real programs.

Note to commenters:  Remember that “<” must be entered as “&lt;”, and “&” must be entered as “&amp;”.  Unfortunately, there’s nothing I can do to fix things if you post a comment with a “<” that gets interpreted as indicating the start of an HTML tag!

Click on image for larger version.

Toronto, May 2011. Pentax ME Super with SMC Pentax-M 50mm 1:1.7 lens, Black’s (Fuji?) ISO 200 film, Nikon Coolscan V.

The main source of this speed penalty is an insistence that the result of a matrix multiply should follow R’s rules for handling infinity, NaN (not-a-number), and NA.  These rules correspond to what happens with ordinary arithmetic operations on modern computers, which follow a standard for floating-point arithmetic in which, for example, 0/0 is NaN.  You might therefore think that nothing special is needed to arrange for matrix multiplies to produce NaNs as required.  However, R does matrix multiplications using the BLAS library, which comes in many versions, some of which may try to speed things up by avoiding “unnecessary” operations such as multiplication by zero — assuming that that will always result in zero.  However, zero times NaN or infinity is supposed to be NaN, not zero.

To ensure that all the right NaNs appear in the result matrix, R currently looks at every element of the operands of a matrix multiply to see if any of them are NaN or NA.  If so, it does the matrix multiplication with a simple set of nested loops in C, which will propagate the NaNs correctly.  Only after verifying that no NaN or NA appears in the operands does R call the BLAS routine for a matrix multiplication.

A comment in the code indicates that whoever wrote it thought that the overhead of this check would be negligible (when no NaN or NA is present), but this is not always true.  If the result matrix has N rows and M columns, and the other dimension of the operands is K, the check will take (N+M)×K operations, whereas the multiplication itself will take N×M×K operations.  The NaN check is actually slower than a multiply, so the overhead of the check will be negligible only if N+M is several times smaller than N×M.  When either N or M is small (say, less than 10), the overhead is substantial.  The worst case is when N and M are both one, which corresponds to a vector dot product operation.  The NaN check may dominate even more if the actual multiply is done in parallel using several processors.

In my first set of speed patches for R, I tried to address this problem by estimating when the check would take longer than the matrix multiplication itself, and if so doing the multiplication with simple loops (avoiding the need for an explicit check). This sped up some operations by large factors, while ensuring that all the right NaNs were produced.  The R Core people didn’t adopt this patch, however, so R’s matrix multiplications are still as slow as ever.

The  latest release of R does have one change, however.  The BLAS routines included with the R release have been changed so that they don’t try to save time by avoiding multiplies by zero.  This has the effect of fixing a bug in 2.11.1 in which, for example, c(1/0,1) %*% c(0,1) produced 1 rather than NaN.  The costly check for NaNs had not actually been a full fix!

This makes the situation in version 2.13.0 rather ridiculous.  Many matrix multiplies are slowed down substantially by a check that is unnecessary if the BLAS routines supplied with R are used.  These checks might make a difference only if some other set of BLAS routines are linked with R. Generally, users will go to the trouble of doing that only if they are looking to improve the speed of their matrix multiplies.  But even the fasted BLAS routine can do nothing to reduce the overhead of the NaN check that is done before it is called!  The final irony is that (as far as I can tell) there will still be some “incorrect” results, when a BLAS optimizes away a multiply of infinity by zero.  (It looks like this could be fixed, however, by simply changing the “isnan” checks to “!isfinite” checks.)

Although my previous patch sped up many operations considerably (eg, vector dot products sped up by something like a factor of six), I now think that the real solution is to just not be so concerned about NaNs. From a statistical viewpoint, does anyone really expect that missing data indicated by NA will propagate through matrix multiplies?  If so, are they also expecting NAs and NaNs to propagate correctly through eigenvalue computations?  At some point, it becomes ridiculous to expect this.  I think that point comes before matrix multiplies.  From a more general viewpoint, the fact that BLAS routines don’t guarantee correct propagation of NaNs is an indication that the community thinks that is less important than speed.  Now that the BLAS supplied with R does propagate NaNs correctly, anyone who is concerned about that can simply use the supplied BLAS. Someone who instead installs an optimized BLAS presumably does so because they want their multiplies to go fast.

I’ve therefore created a modified version of R that omits the NaN check. Also, rather than always call the BLAS general matrix multiply routine, DGEMM, I call the routines for vector dot products (DDOT) and matrix-vector products (DGEMV) when the dimensions of the matrices are suitable.  Calling these more specialized routines sometimes gives faster results with the BLAS supplied with R.  The modified version of the R matrix product routine is here; the original from 2.13.0 is here.

Below, are the factors by which this modified version speeds up matrix multiplies, on my new workstation with an Intel X5680 processor, running at 3.33GHz:

The speed ups seem to me to be quite worthwhile. This is with the BLAS supplied with R. I expect that the speed ups with an optimized BLAS would be even larger.

Note again that when using the BLAS supplied with R, there is no problem with NaNs not being propagated. There might be a problem with some other BLAS. However, it is difficult to see why an optimized BLAS would check whether an element of a vector in a vector dot product is zero, since it will be used in only one multiply.  Checking whether an element of a matrix in a matrix-vector product is zero seems similarly counterproductive.  Any problem is therefore probably confined to vector elements in a matrix-vector product or matrix elements in a matrix-matrix product that isn’t a matrix-vector or vector dot product. So if one were to insist on doing a NaN check, one might be able to look only at those elements.

I’ve been running these tests on my new workstation, which has a six-core Intel X5680 processor, running at 3.33GHz.  Unfortunately, it’s clear that thing runs somewhat slower when you use all the cores at once, so for consistency one needs to do the speed tests using just one core.  (Or one needs some more elaborate, and unclear, protocol for testing the speed of R in a muticore environment.)  I haven’t figured out how to get Red Hat Linux to compile 32-bit applications yet, so all the tests are in a 64-bit environment.

I’ve started with comparing the speed of R-2.13.0 with and without functions being compiled, and with comparing R-2.13.0 (without the compiler) to R-2.11.1, which was the last release before some of my speed improvements were incorporated.  A plot of the results is here.

Looking first at the effect of the compiler in R-2.13.0, one can see that for programs that do simple operations in loops, the compiler can speed things up by up to a factor of five, though the speed-up is often less than a factor of two, and in one strange case (a very simple for loop) the compiler slows things considerably.  As one would expect, there is no speed-up for programs dominated by large operations such as matrix multiplies.  There is also little speed-up when operations like matching arguments dominate.  There’s a modest speed-up for the vector arithmetic tests, which may be related to storage allocation.

Looking at R-2.13.0 versus R-2.11.0, one can see modest speed-ups for programs doing simple operations, which I believe is due to my improvements to “for” and to construction of argument lists.  There are also major improvements to some operations like “transpose”, which are also all due to modifications I introduced, with the exception of the improvement for matrix multiplies, which I believe is due to recent changes to the BLAS, which eliminate some special checks for zero, probably motivated by concern for proper NA/NaN propagation.  (My proposed modifications to matrix multiplies can produce a much larger improvement, but were not incorporated.)

Many of my other speed improvements have also not been incorporated into the released version of R.  I’m currently updating them for R-2.13.0, and adding some new speed improvements.  I hope to release them soon.

I expect that the speed-ups from these improvements will often be comparable to that obtained from using the compiler.  Indeed, in some cases they will be the same improvements — the compiler includes some optimizations that can just as easily (or more easily) be done in the interpreter.  For instance, the interpreter currently allocates new space for TRUE or FALSE for the result of every comparison or logical operation.  I came up with a simple modification to just allocate TRUE, FALSE, and logical NA once, and then re-use them as needed.  I then noticed that the compiler does something similar.

Other speed-ups will be different, however.  It will be interesting to see the combined effect of using both my speed improvements and the compiler.

UPDATE: I’ve released a new version of these speed tests, which fixes some glitches, adds some new tests, and improves the appearance of the plots.  You can get the new version (and new plots comparing 2.13.0 with and without compilation and 2.11.1 versus 2.13.0)  here.

MCMC Using Ensembles of States for Problems with Fast and Slow Variables such as Gaussian Process Regression

I wanted the title to mention all three of the nested ideas in the paper.  Actually, I wasn’t able to fit in a fourth, most general, idea, of MCMC methods based on caching and mapping (see here).   Here is the abstract:

I introduce a Markov chain Monte Carlo (MCMC) scheme in which sampling from a distribution with density π(x) is done using updates operating on an “ensemble” of states. The current state x is first stochastically mapped to an ensemble, x⁽¹⁾,…,x^(K). This ensemble is then updated using MCMC updates that leave invariant a suitable ensemble density, ρ(x⁽¹⁾,…,x^(K)), defined in terms of π(x⁽ⁱ⁾) for i=1,…,K. Finally a single state is stochastically selected from the ensemble after these updates. Such ensemble MCMC updates can be useful when characteristics of π and the ensemble permit π(x⁽ⁱ⁾) for all i in {1,…,K} to be computed in less than K times the amount of computation time needed to compute π(x) for a single x. One common situation of this type is when changes to some “fast” variables allow for quick re-computation of the density, whereas changes to other “slow” variables do not. Gaussian process regression models are an example of this sort of problem, with an overall scaling factor for covariances and the noise variance being fast variables. I show that ensemble MCMC for Gaussian process regression models can indeed substantially improve sampling performance. Finally, I discuss other possible applications of ensemble MCMC, and its relationship to the “multiple-try Metropolis” method of Liu, Liang, and Wong and the “multiset sampler” of Leman, Chen, and Lavine.

I’ve also posted the programs used to produce the results.  These haven’t been tested much beyond their use for the paper, but I hope to incorporate them into a general MCMC package in R (also including programs accompanying my review of Hamiltonian Monte Carlo).   That’s my next project to do in whatever time I have available after teaching, administration, and a three-year-old daughter, along with more efforts to speed up R, so the that this MCMC package won’t be too slow.