Wednesday, February 17, 2010

The Null Hypothesis

This post is a follow-on to discussions on the use by Mann et al (PNAS, 2008) of the Lake Korttajarvi varve proxies described by Tiljander et al (2003), at Lucia's Blackboard and at Michael Tobis' Only In It For The Gold.

Nick Stokes asserted,
Mann fixed Tiljander in the SI of the 2008 paper. He did an analysis without it. If the calibration of the proxy doesn’t work, that’s all you can do.
This follows in the footsteps of Gavin Schmidt's analysis at RealClimate.org --
[Response: This issue was discussed ad nausem at Stoat - bring it up there... There is no other possible reconstruction that would use the proxy in another orientation. It is either in the way it was, or it isn't included at all. Both options were published together in the PNAS paper. No correction needed... ]
Emphasis added.

In this post, I argue that understanding the implications of Mann et al.'s use of the Lake Korttajarvi varve series is essential to a reasoned interpretation of their work.


[Update 9 March 2010: Footnote moved to end, images added, footnote legends improved.]

In order to undertake an informed discussion of Mann et al. (2008), one has to have a clear understanding of the presentation and use of the underlying data, most especially the Lake Korttajarvi proxies. No correction needed should be scorned as sloppy thinking and bad science. Correction required! is a much better slogan.

Discussions with Mann's defenders about his use of the Lake Korttajarvi proxies develop a circular quality, reminiscent of the screenplay of Groundhog Day.

Mann et al. (2008) is fine as is.
X is right!
Well, I grudgingly concede that you (may have) proved that X is wrong.
If I were Mann, I would fix X.
Mann should fix X.
Come to think of it, figures like his SF8a prove that X doesn't make a difference to his results.
Thus the impact of any possible correction has been shown to be trivial.
Mann refused to fix X, fixing X would be trivial, thus No Correction Needed.
Mann need not fix X.
Mann et al. (2008) is fine as is.

Let's go back to the beginning. Why did Mann and his coauthors perform this work? Why did the editors and peer reviewers of the high-prestige, high-impact journal The Proceedings of the National Academy of Sciences of the USA deem this work worthy of publication?

The frame of reference of Mann '08's authors was:

"We have a good system for extracting temperature signals out of proxies to generate paleoclimate reconstructions. Let's go well beyond bristlecone pine tree rings, and employ other proxies, as well."

This led to:

"The Lake Korttajarvi varve series looks like a promising paleotemperature proxy. Are its data series valid? Do the concerns of the original authors disqualify them? As careful scientists, we'll tackle this question by performing analyses both with and without these four series (Supplemental Figure 8a). We show that the reconstruction with the Lake Korttajarvi series are nearly identical (later revised to "very similar") to those without (see Footnote). We conclude that the Lake Korttajarvi proxies tell a story that is consistent with those told by the other proxies we've employed. The addition of data series of another provenance (lakebed sediments) and from another part of the world (central Finland) strengthens our confidence in our overall reconstruction of past climates."

- - - - -

Readers of Mann et al. (PNAS, 2008) have important information that the paper's authors lacked when they submitted it.

We know that the Lake Korttajarvi series are among the worst possible data to add into a temperature reconstruction.

1. Since they are uncalibratable, all four are grossly miscalibrated to the instrumental record.

2. Two and probably three of the proxies were added in to the reconstruction algorithim in an upside-down orientation. Whatever "warmer" information they contain was misread as "colder," and vice versa!

3. The authors thought that their proxy-qualification test (done by splitting the 1850-1995 instrumental-record period in two) showed that the Lake Korttajarvi proxies were valid. We now know that those results were entirely spurious.

With hindsight, we can see that Mann and his coauthors completely misinterpreted the Lake Korttajarvi proxies. Figure SF8a in no way means what the authors thought that it meant.

Instead, the use of the Lake Korttajarvi proxies constitutes a powerful test of an extremely important null hypothesis.

Null Hypothesis: "The combination of Mann et al. (2008)'s proxy selection protocol and computational algorithm fails to produce a valid temperature reconstruction."

Specifically: "We have derived a paleotemperature reconstruction that we believe has merit ("NH CPS minus 7"). If we pollute the input data with spurious information to generate a pseudo-reconstruction ("NH CPS"), the pseudo-reconstruction will deviate greatly from the genuine one. Falure to show such deviation would demonstrate that the algorithm we used to construct the genuine reconstruction is refractory to corrupt, meaningless input. In that case, we would be required to accept this null hypothesis."

This null hypothesis should be accepted!

I am sure that there are avenues that Mike Mann can explore, where he attempts to rebut this interpretation of his reconstruction. He can run down the reasons why this formulation of a Null Hypothesis is incorrect, or why the upside-down use of the uncalibratable Lake Korttajarvi varve series does not sink his proxy-selection or data-analysis methodology.

By all means, Mann should do so. When he submits his corrected version of Mann et al. (PNAS, 2008) as a belated response to McIntyre and McKitrick's Comment (PNAS, 2009).

But returning to the extended Groundhog Day arguments about "No correction needed" have the effect of excusing Mann and his coauthors from facing up to their responsibilities as scientists.

* The Methods should be correct, certainly as far as how key inputs were used!

* The Discussion should squarely discuss the reconstructions that Mann et al generated and presented, in light of the corrupted series that they incorporated.

Mann and his coauthors should not be given a pass on using Proc. Natl. Acad. Sci. USA as a vanity press.
____________________


Footnote -- The relative influence of the four Lake Korttajarvi proxies (and three others) can be see by contrasting the "NH CPS" (Northern Hemisphere Composite Plus Scale) trace with the "NH CPS minus 7" trace on the three Supplemental Figure 8a's that Mann has successively released.

Original SF8a -- This is Supplemental Figure 8a as uploaded with the paper on 2 Sept. 2008. It remains unchanged (downloaded from the PNAS.org website on 9 March 2010). The "original NH CPS" trace (green) is virtually identical with "NH CPS minus 7" trace (black).


Once-revised SF8a -- This corrected version was uploaded to the 'Meteo' website at Penn State on 1 Dec. 2008 (it is not at PNAS.org). Text in a nearby directory reads: "UPDATE 1 December 2008: Supplementary Figure S8a had a small error due to improper calculation of the validation statistics." In this figure, "original NH CPS" (green line) is almost superimposable with "NH CPS minus 7" (black line), with the exceptions of 1020-1100, 1120-1200, 1350-1390, and 1630-1640.

Twice-revised SF8a -- This re-corrected version was uploaded to the 'Meteo' website at Penn State on 4 Nov. 2009 (it is not at PNAS.org). Text in a nearby directory reads: "UPDATE 4 November 2009: Another error was found in the corrected Supplementary figure S8a from December 2008: The previously posted version of the figure had an error due to incorrect application of the procedure described in the paper for updating the network in each century increment. In the newly corrected figure, we have added the result for NH CPS without both tree-rings *and* the 7 potential 'problem series'..." "NH CPS" (black line) is virtually identical with "NH CPS minus 7" (green line), except 1610 and 1640-1680.

No comments:

Post a Comment