Answers to Applied PhD Comprehensive Question #2

2008-10-13 at 12:21 pm 6 comments

I’ve been busy with teaching, so I’m only now getting around to posting the answers to the second applied statistic comprehensive exam question that I posted (here).

Originally, I’d thought that posting the answer would be simply a matter of extracting the answer I’d already written before.  But looking it over before posting, I noticed that I’d made an additional unintentional error in the analysis that the question asks you to critique! Of course, one could say that the more errors the better, but I did need to update my answer to reflect this.

The error is that in the first analysis presented, I had intended to include interaction terms between treatment and covariates such as sex in the regression shown.  Due to some sort of momentary brain failure, I instead just put these covariates in by themselves.  When writing up my answer later (which students who wrote the exam got), my mind was set on the idea that I’d put in interaction terms, so this error wasn’t reflected in that answer.  I don’t think this had any significant effect on the marking, fortunately, since the comments on the later analyses aren’t really affected.  It does show how easy it is to keep seeing what you expect to see.

Here’s the PDF file with the answers.  The questions are included as well, so no need to refer back to them.

Entry filed under: Statistics, Statistics - Nontechnical.

Design Flaws in R #3 — Zero Subscripts Downtown with Sky

6 Comments Add your own

  • 1. ZBicyclist  |  2008-10-15 at 1:46 pm

    How many of your students pointed out that the “sex” effect in the first analysis should have been an interaction to draw the conclusion the researchers made?

  • 2. Radford Neal  |  2008-10-15 at 2:24 pm

    It was a while ago, so I don’t recall their answers in detail, but if any pointed this out, they didn’t do so emphatically enough for me to actually realize it at the time! Either they too saw what they expected to see (not too surprising in the heat of an exam), or I was still seeing what I expected to see so strongly that any comment they made failed to register…

  • 3. Keith O'Rourke  |  2008-10-16 at 1:41 pm

    I had presummed the analysis had purposely been done incorrectly – and was at a loss as to how to answer the questions with regard to the other studies…

    But even with your as you say less than ideal fix (assuming no main effects in the presence of an interaction and disregarding covariates fit other than sex and treatment) to extract seperate treament effects the non-comparability of age raises issues as to whether any comparisons should be made

    I liked the presence of more than one study in the question very much but think it would be much better if the first other study is comparable (not sure how many grad student would do well even in that case)


  • 4. Radford Neal  |  2008-10-16 at 2:25 pm

    The “non-comparability” of age was a very deliberate feature of the question, and surely is an issue that can very well arise in practice, so one can’t just give up and say that nothing can be said. The main point of asking about the second study was to see if the students realize that the small sample size means that failure to reject the null hypothesis doesn’t mean it’s true (or even close to true). For the third study, the main point is that there’s no contradiction between the first study finding clear evidence of an average effect, and the third study finding little or no effect for a sub-population, especially given that there is evidence from the first study (eg, the histograms) that there may be different effects in different sub-populations.

  • 5. Keith O'Rourke  |  2008-10-16 at 3:12 pm

    Yes, the non-comparible studies should remain but a comparible study added

    p.s. doesn’t the evidence from the histogram depend on the scale of outcome chosen?

  • 6. Nigel Goodwin  |  2012-05-19 at 10:58 am

    I’ve never formally studied probability and statistics (I was originally a theoretical physicist), but I have done a lot of regression.

    My answer would have been – the histograms look very similar, but the treatment has higher variance, this indicates that further work is needed.

    The R2 or adjusted R2 is a joke. I wouldn’t trust anything the regression says.

    T or P or anything tests on interaction terms is a joke. You need to do much more analysis.

    Job done. Fail (the drug and my future career as a statistician).

    [I say this last point because, in my field of reservoir uncertainty, people do this kind of simple analysis and find a T test for the effect of porosity x permeability, and try to make conclusions, it is utterly flawed, particularly when they only have 100 samples. The scope for overfitting and spurious effects is overwhelming.]

    But I like the questions – makes the students think, rather than just churn the maths.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


October 2008

Most Recent Posts

%d bloggers like this: