Here are the R program and plots produced by it that you sent by email. I haven’t absorbed the plots entirely, but it’s certainly the case that the inconsistency of the MLE is due to a few (one, typically) data points dominating.

Yes. In that case, the data space would be finite (we can ignore the infinity in the big direction), and with enough data, the probability of each of these possible data values would be very well estimated. Values for the single model parameter map to distributions over these finite number of data values, in a continuous and probably one-to-one fashion, allowing the parameter to be well estimated by maximum likelihood.
However, you shouldn’t conclude that results about inconsistencies of MLEs aren’t relevant in practice, since data is always rounded. The practical effect of an inconsistent MLE is that the MLE also isn’t very good for finite amounts of data. Even if the data is rounded, so the MLE is consistent, this bad performance for finite amounts of data may remain. In a problems with higher-dimensional data, even fairly coarse rounding will still produce a huge number of possible data values, so that the finiteness may not be of any practical relevance.

Keith

Occured to me this morning if one took account of observations being actually discrete rather than truly continuous and replaced the density with the integral from obs – e to obs + e the inconsistency would go away – if e was big enough?

cheers
Keith

I think the code got sort of destroyed in your comment by the blog software interpreting less-than and greater-than signs as HTML commands. To get a less-than through, you have to use ampersand, “lt”, semicolon. I think…

Here’s a try: < Worked?

This example is interesting to me because even though we know the observations were generated with a single (common) parameter most individual observation likelihoods support .6 but a few support (different?) small values around zero – and these few out weigh the majority… so the parameter value most likley to generate this set of observations is “misleading”.

Simple modifications the the plot.lik code to get these plots are given below (on the log scale).

cheers
Keith

function (x,…)
{
grid 0 & x<3],seq(0.01,3,by=0.01)))

ll <- rep(NA,length(grid))
for (i in 1:length(grid))
{ ll[i] <- log.lik(x,grid[i])
}

mlv <- max(ll)
mle <- grid[ll==mlv][1]
max.ll <- round(mlv/log(10))

#ll <- ll – max.ll*log(10)
#lik <- exp(ll)
mlvi=(1:length(grid))[ll==mlv]
ll <- ll – ll[ll==mlv] + 2
lik <- ll

plot(c(-0,2),c(-8,max(lik) + 2),type=”n”,
xlab=”data / parameter”,
ylab=paste(“likelihood (x 10^”,max.ll,”)”,sep=”"),
…)
points(x,rep(0,length(x)),lty=19)
lines(grid,lik)

title(paste(“Likelihood -”,length(x),”data points,”,
“MLE approximately”,round(mle,4)))
iol=NULL
for(k in 1:length(x)){
for(ki in 1:length(grid))
iol[ki]=log.lik(x[k],grid[ki])
lines(grid,iol – iol[mlvi] + 2/length(x),col=2)}
}

In this example, parameter values just above zero are the source of the problem, but of course other values might be the problem for other models. By “finite parameter space” I meant that that there are only a finite number of possible values for the parameter (so the parameter space isn’t an interval of the real line, for instance). An approximation with a finite parameter space will be good for all values of the parameter if slight changes in parameter values don’t have huge effects. In the model of this post, that’s not true, because even though the density never becomes singlular (so there aren’t any infinite density values), it does become increasingly peaked as the parameter gets closer to zero.

A sample size of 30 can be large enough to well represent the true density. You may draw a histogram to see this. Actually as illustrated by Radford, it will be even worse with more samples.

“the infinity induced by the density concentrated at zero” is a (infinite) value of the likelihood function at a parameter point (close to zero). The value of the likelihood function at a parameter value equals to the product (or sum in log-scale) of the values of the density function at each data point with that parameter value. Therefore, for a given parameter value, how large the likehood function is is a combined effect from all the data points through their density function. However, if one data point has some type of dominant contribution (e.g. infinite density value), averaging over even a large number of values may not be able to remove atypical phenonimena like “infinity”.

My opinion though.