Three comments threads at William M. Connolley’s blog Stoat come closest, in my opinion.
Although a couple of obstacles became evident over the course of the three conversations there (Oh dear oh dear oh dear oh dear, Tiljander, and Tiljander, again).
One is that WMC often responds to critics by inserting remarks into the middle of their arguments, making those arguments harder to follow. Another is the two-edged sword represented by his liberal comment-truncation and deletion policy: the tactic can keep conversations on track, but it can also cause the facts and reasoning marshalled by critics to appear weaker and more disjointed than is actually the case.
This weekend, I reviewed the comments thread of Oh dear oh dear oh dear oh dear, revisiting some of the key points debated there. For context, the post itself is a mocking criticism of Roger Pielke, Jr., contending that Pielke “doesn’t’ understand” Mann et al’s use of the Lake Korttajarvi varve proxies. Most interesting is that WMC is an accomplished software engineer and climate modeler who has written extensively about sea ice and warming trends in the Antarctic, and who has authored posts at the flagship AGW Consensus blog RealClimate.org defending and defining that Consensus--prominent ongoing topics at Stoat.
An abridged and annotated version of that post's comments thread follows the "read more" break. For clarity, I have moved WMC’s interspersed comments to the end of each comment. I have also added some of my own thoughts; these are limited to the paragraphs that begin with the text "AMac 12/7/09".
My hope is that this discussion will help readers evaluate my claims about Mann et al (2008)’s use of the Lake Korttajarvi proxies. In my opinion, there are three fundamental questions:
1. Did Mann et al mistakenly use some of the proxies in an upside-down fashion, relative the meaning that Tiljander et al (2003) assigned to the data? (Simple question, easy answer: Yes, they did. The “xraydenseave” and “lightsum” series are inverted. The “thicknessmm” series is arguably upside-down, as well. “darksum” is right-side-up. Perhaps you are inclined to accept the Pro-AGW-Consensus argument that a data series cannot be used upside-down? Please read the Jarvykortta River proxy thought-experiment, and comment there.)
2. Did Mann et al mistakenly calibrate the Lake Korttajarvi proxies to a time period where Tiljander cautioned that the climate signal in the data was overwhelmed by the effects of nearby human activities? (Simple question, easy answer: Yes, they did.)
3. Are there additional fundamental problems with the logic, mathematics, and statistics that Mann et al employed in their use of the proxies to construct the multi-century temperature anomoly graphs that are at the heart of the paper? (Answer: Yes, there are. Unlike the first two issues, this one is complex. It is only touched on in Oh dear oh dear oh dear oh dear.)
WMC's post and certain illuminating comments follow.
Oh dear oh dear oh dear oh dear
Posted on: October 27, 2009 6:46 PM, by William M. Connolley
Please don't force me to write another of these, I'll run out of "oh dear"s.
The issue is RP Jr venturing into areas of climate science he doesn't understand (see losing the plot for the last one I remember) and dragging Cruel Mistress along behind, though to be fair CM doesn't fully commit herself.
AndrewT has already explained the truth to Roger, but it doesn't look like he wants to know it :-(.
[AMac 12/7/09: I snipped a few "off-topic" concluding sentences from the original post.]
Comments to "Oh dear oh dear oh dear oh dear"
Comment 9 | Posted by AMac | October 27, 2009 10:56 PM
I'm a lay person who was intrigued by Roger Pielke's account and the discussion in his comments. I pulled the Mann et al. (2008) PNAS paper and Supporting Information, which led to a some questions, and considerable puzzlement. I posted them as a two-part comment over there, #35 and #37.
To me, the heart of the matter seems to be this. Mann et al. used a series of 15 time-series data sets (Fig. S9) to derive a “Temperature Anomaly vs. Time” graph for the Northern Hemisphere (Figs. 2 and 3). It turns out (Mcintyre claims that) that Mann et al. made an error in handling one of the 15 proxies (Tiljander's X-Ray Diffraction measures of the Lake Korttajarvi varve series). Mann et al. assumed that lower XRD values signify cooler temperatures, whereas Tiljander and others assign lower XRDs to warmer temperatures.
[WMC #1]
In addition, Tiljander states that the four Lake Korttajarvi records (XRD and three other varve characteristics) are unreliable after about 1720 because of nearby human activity. Peat ditch-cutting in 1930 and bridge construction in 1967 are the causes of the most prominent recent variances in three of the four records, clearly visible in Fig. S9.
[WMC #2]
How is it possible to remove these four records (and three others), leaving eight proxy series for the Northern Hemisphere, and end up with a Temperature Anomaly reconstruction that is indistinguishable from the one constructed from all fifteen proxy series (Fig. S8a, green vs. black line)?
[WMC #3]
Mann seems to be saying that reversing one of the fifteen proxies (mistakenly correlating higher XRD with higher instead of lower temperature) doesn't alter the temperature reconstruction. How can this be? If the varve XRD proxy is inconsequential, why include it at all?
[WMC #4]
Could any of the other 14 proxies have their polarity reversed without effect? How about two? three?...
[WMC #5]
This doesn't make sense to me.
[WMC #6]
WMC inline responses:
#1 - OK, yes. This is the assertion that the series is “upside down”. Mann et al. (and I) claim this doesn't matter.
#2 - This is a different - though important in its own right - matter. It may be peripherally related.
#3 - That would require various bits of analysis which I can't readily do to answer. But I can offer a hand-wavy answer. Suppose the lake sediments have been so affected by recent (non-climatic) changes that they are just random noise. then their correlation against recent temperature will be essentially zero. So their contribution to the reconstruction will be very small. You can (I speculate) extend this argument to a large number of proxies: you can throw in any number of series with no idea of their correlation to temperature, and the method will (on average) sort them all out. Of course, if *none* of your records usefullly records temperature you won't get anything interesting out.
#4 - No, you've misunderstood. Its not that its inconsequential, just insensitive to sign. See my helpful formula in one of the comments higher up.
#5 – All.
#6 - And yet it is not that difficult. Maybe I need to blog this.
AMac 12/7/09: Response #1 suggests that WMC has waded into this discussion without understanding the gravity of Mann et al’s problems with the Lake Korttajarvi proxies. (At this early stage, I didn’t, either.) In response #2, WMC is already failing to defend Mann et al’s calibration of the varve series. This is (or should be) fatal to their use, even if all of them had been used right-side-up. The point about corrupt proxy data leaving the temperature traces in Mann et al Figure S8a unaffected speaks to the other major defects in Mann et al’s paleoclimate reconstructions. In responses #4 and #5, WMC is completely incorrect: using a thermometer upside-down is fatal to temperature reconstructions.
Comment 10 | Posted by: Rattus Norvegicus | October 28, 2009 12:21 AM
AMac,
I suggest you take a look at figure 1 in the actual paper. This shows the distribution of proxies in the data he used. It was over 1200 in total more than 400 passed the local temperature screen. Dropping 4 or even 8 series from this (series which Mann admits in the SI to be problematic) reconstruction would obviously have little effect on it, contrary to McI blowing this up into some case which should be heard in the war crimes court at the Hague.
As far as how the two recons are virtually identical, in the modern period hundreds of proxies are in use and so the effect of these problematic proxies is small. The larger effect is probably seen in the error bars associated with values in the 9th and earlier centuries, take a look at Figure S10 which shows the effects better due to the expanded scales used.
WMC response: I'd rather this didn't broaden out into the usual discussion of the validity of the reconstructions. This should focus on the narrow point: does the sign of the proxies matter. So far the only serious objection I've seen is “it doesn't matter for Regem but does for CPS”. Anyone with an informed view on that is welcome to comment.
Comment 11 | Posted by: AMac | October 28, 2009 1:46 AM
Thanks, Rattus Norvegicus! I see that I'm asking about a smaller issue (the 15 screened proxies available for the NH CPS reconstruction back to the 9th century and earlier, end of pg. 13255 ff). Your point on the broader issue is that there are much, much larger numbers of proxies available for more recent times (red symbols in Fig. 1).
McIntyre claims that all four of the Lake Korttajarvi varve proxies had their signs reversed in Mann et al (2008), not just the XRD one.
Thus, I remain perplexed by Fig. S8a. That depicts the NH Land Temperature Anomaly reconstructions, performed (1) with the full proxy network (green line), and then (2) with the eight proxies remaining after the removal of seven potentially problematic series (black line).
The full proxy network would be the fifteen NH proxy records that pass the screening procedure back to AD 818 or earlier, shown on Fig. S9. Of the seven removed for the black line, four are the four flipped-around Lake Korttajarvi series. Yet the two reconstructions look almost identical, from 200 AD through to 2000 AD.
With so few proxy series, shouldn't this removal have led to a very different-looking curve?
(I'll cross-post this at Pielke's blog.)
AMac 12/7/09: In this comment, I incorrectly stated that all four Lake Korttajarvi proxies were used upside-down with respect to the orientations proposed by Tiljander et al. Actually, two were upside-down, one is ambiguous, and one is right-side-up.
Comment 12 | Posted by: andrewt | October 28, 2009 1:49 AM
AMac, yes if a method is insensitive to sign, you should be able to reverse the polarity of any or all of the inputs and the output should remain unchanged. As Marcus says above, this does seem to raise other questions - which sadly you might actually need to know something about paleoclimatology to consider.
In truth I was more fascinated by Roger wading into paleoclimatology after his recent railing that the professionally-unqualified should not be debating his work.
AMac 12/7/09: AndrewT is correct, but only in detail. In the case of two of the Lake Korttajarvi proxies, the outputs may indeed remain unchanged—but they are inverted with respect to Tiljander’s assignments! Again: reading a thermometer upside-down is fatal to a climate-reconstruction project.
Comment 14 | Posted by: AMac | October 28, 2009 7:23 AM
andrewt, thanks.
> if a method is insensitive to sign, you should be able to reverse the polarity of any or all of the inputs and the output should remain unchanged.
As a general statement, this is certainly true, and it seems to apply to the four upside-down Lake Korttajarvi varve proxies and the reconstruction in Figure S8a. But that's what puzzles me: what, exactly, does Figure S8a show?
WMC response: Cut. Sorry, perfectly sensible comment, but not on the (restricted) topic.
AMac 12/7/09: The full text of the comment is at this cross-post.
Comment 15 | by road | October 28, 2009 7:23 AM
You guys are missing the point here.
Yes, the sign going into a multivariate regression doesn't matter. But the sign coming out certainly does. And if the correlation coming out has a sign opposite to the a priori physical meaning, then we have a problem. That means the correlation is aphysical and spurious.
[WMC #1]
The CPS method used by Mann (one of two methods he used) is not a multivariate regression. As described by CA “CPS is little more than averaging and shouldn't take more than a few lines of code. You standardize the network - one line; take an average - one more line; and then re-scale to match the mean and standard deviation of your target - one more line.”
So, the sign clearly does matter.
[WMC #2]
Guys, this one is a no-brainer.
WMC inline responses:
#1 - The sign going in is the point at issue here. If the sign going in is wrong, then the correlation has the same absolute value but the wrong sign, which means the series gets added back in to the reconstruction with the sign wrong twice, ie correctly. If by “the correlation is aphysical” you mean, of the wrong sign, then you are correct. But this is irrelevant.
#2 - I'm unsure what you mean by that. You're clearly wrong on your point 1 - are you applying this “no-brainer” comment to yourself? On point 2 there is unclarity, but simply relying on CA to tell you the truth seems unwise.
AMac 12/7/09: Road’s analysis is correct. WMC’s rebuttal is nonsensical. Reading a thermometer upside-down does matter.
Comment 16 | Posted by: Peter | October 28, 2009 7:52 AM
Point #15 clears this up quite nicely. Since Kaufman has issued a Corrigendum in which the upside down series is acknowledged, is he wrong or is Mann?
WMC response: He is right and Mann is right.
AMac 12/7/09: WMC’s response is again nonsensical. Kaufman followed Mann’s lead and read a thermometer upside-down. When this was brought to his attention, he corrected the error. Kaufman and Mann cannot both be correct.
Comment 17 | Posted by: jeff id | October 28, 2009 9:30 AM
#8 William,
CA has replicated the whole CPS portion of the study. In the case of CPS the ambiguous proxy is flipped according positive or negative correlation. I'll dig through the code later if I get a chance and point to the exact line where the flipping occurs or doesn't. When CPS is finished you have a multiplier vector 'a' times array 'Ta' for each proxy. The sum of the array after multiplication times a is the result - flipped proxy!
In the case of EIV the process is a 'best fit' of proxies to the temp record - iterative multivariate regression through expectation maximization.
If you took 100 downslope tiljanders + a small amount of noise and fit them to an upslope temp record, they would mostly flip upside down matching the temp rise with a probability that some were used upside right. The net sum would give a best match to the temperature upslope even though 100 percent of the data had the original Tiljander downslope initially.
Of course despite the match to instrumental records, the important historic result would have very little to do with temperature and were the proxy related to temperature, the historic portion would represent anti-Tiljander temperature after averaging.
It's really a very very dead issue.
AMac 12/7/09: Jeff Id’s analysis is correct and has stood the test of time.
Comment 18 | Posted by: David | October 28, 2009 10:19 AM
“If by ‘the correlation is aphysical’ you mean, of the wrong sign, then you are correct. But this is irrelevant -W"
How can it be irrelevant that a proxy reconstruction is inverting constituent proxies with a known (or presumed) physical correlation to that which the reconstruction is attempting to reconstruct ?
WMC response: Cut. Answer: read the stuff above.
AMac 12/7/09: There is no adequate rebuttal to David’s points, earlier (or later) in this thread.
Comment 19 | Posted by: Jason | October 28, 2009 10:28 AM
William, are you seriously defending Mann's use of Tijander with the given orientation?
WMC response: Trimmed. If you've read what I've written before, you know the answer to that.
Comment 23 | Posted by: AMac | October 28, 2009 12:40 PM
The validation period for the 15 screened NH CPS proxy series was 1850-1995 (pg. 13254). Tiljander states that natural signals in the Lake Korttajarvi varve proxies are disrupted after ~1720 by nearby human activities (SI, pg. 2). Thick, mineral-rich varves due to ditch-cutting in 1930 and bridge-building in 1967 are prominent in Fig. S9.
The CPS method was applied in three different ways. Full CPS (calibration 1850-1995), Early (1850-1949) Calibration [Validation 1950-1995], and Late (1896-1995) Calibration [Validation 1850-1895] (pg. 13254).
The Full, Early, and Late 15-proxy reconstructions have the same general shape from 400 to 1850, though there are periods of divergence prior to 1700 (Fig. S11a).
For the varve proxy series, all three CPS calibration approaches appear to have led to the correlation of the deposition of thick, mineral-rich varves with higher temperatures.
Thus, it would appear that the four Lake Korttajarvi proxies contributed a “warmer” signal to the 400-1850 CPS reconstructions in the decades when varves were thicker and more mineral-rich. And a “cooler” signal when varves were thinner and included more organic material. (See orientation of Tiljander proxy plots in Fig. S9, and CPS curves in Figs. 2, 3, S7, S8, S10, & S11.)
This appears to be the opposite interpretation to the one offered by Tiljander, who correlates thinner, more organic-rich varves with higher temperatures. Kaufman accepts Tiljander in his 10/9/09 correction.
If this description is correct, it would appear that there is a “sensitivity to sign” in the CPS method of analyzing climate proxies to derive reconstructions of hemispheric temperature anomalies.
AMac 12/7/09: This description is correct.
Comment 25 | Posted by: William [Connolley] | October 29, 2009 4:51 AM
Commenting on my own post... how gauche. Anyway, over at CM is an intersting and perhaps enlightening comment by Bender, which may indicate that the entire debate is shifting. I don't know if this is merely B's viewpoint, or if the CA types have realised that the “upside down” charge in its original form won't fly. I'll copy my reply, which I think quotes B's main points:
The substantive issue is how does Mann’s code treat a proxy when its relationship with temperature changes as you move from the calibration phase into the reeconstruction phase. Aha! Thank you. This is the first time someone has made a coherent argument over this (perhaps it has been said before but lost in the noise, if so my apologies for missing it).
I would answer that such a proxy is simply useless. Getting the sign of the overall series right would not make it useful. A proxy with the properties you describe should not be used.
And it’s not a statistical issue; it’s a coding issue I disagree. It is a data-source issue.
Mann should have investigated more thoroughly once he’d seen the McIntyre complaint – not sure about that. McI’s complaint (assuming we’re talking about the same text) was: “Their non-dendro network uses some data with the axes upside down, e.g., Korttajarvi sediments, which are also compromised by agricultural impact (M. Tiljander, personal communication), and uses data not qualified as temperature proxies (e.g., speleothem δ13C). “ If he meant what you said, he could and should have said so.
Incidentally, “which are also compromised by agricultural impact (M. Tiljander, personal communication)” is an interesting phrase – this appears to imply that the compromise wasn’t clear without pers comm.
AMac 12/7/09: In other comments (search ClimateAudit.org), bender has unambiguously asserted (1) that Mann et al used Lake Korttajarvi proxies in an upside-down manner, and (2) that it would be impossible to calibrate the proxies to the instrumental record. In her 2003 paper, Tiljander discusses the overwhelming non-climate signals in the varves in recent times--some of these cautions are quoted by Mann et al (2008) under “Potential data quality problems” (Supporting Information, page 2). Knowledge of these issues is not limited by "personal communications."
Comment 26 | Posted by: Jason | October 29, 2009 10:36 AM
William, you're missing the basic point here.
A temperature reconstruction is _supposed_ to be a logical argument which shows that, as long as a certain set of assumptions hold, past temperatures may be characterized by a calculated curve.
The problem with Tiljander is not that Mann somehow miscalculated. The problem is that the basic assumptions which give Mann's calculations meaning have been violated.
Mann's logical argument assumes that the relationship between Tiljander and local temperature has been uniform during the period for which its signal has been used. This is plainly (and in this case spectacularly) false.
During the period for which we have instrumental temperature readings, the Tiljander signal is dominated by non-temperature related anthropogenic activity (building bridges, etc.). During earlier periods it is believed (by the people who collected the data) to be correlated with temperature in the opposite direction of what Mann has used.
The end result, is that warmer MWP temperatures in Finland cause Mann's temperature reconstruction to go down, and visa versa.
To the extent that Mann would attempt to use his reconstruction to make arguments about current temperatures compared to MWP temperatures (unprecedented in the last 2000 years kind of arguments), his argument is logically flawed.
In summary, the people who are saying that the proxy is upside down are NOT saying that Mann made an arithmetic error. They are saying that, due to modern artifacts, several (heavily weighted) components of Mann's calculation have a negative correlation with temperature.
If Mann's curve does not represent past temperature, then its not a temperature reconstruction.
AMac 12/7/09: A consise, logical, and correct argument.
Comment 28 | Posted by: bernie | October 29, 2009 11:29 AM
William, you wrote:
“The proxy is correlated against the instrumental record before being amalgamated into the global record. So T_glob = avg(Proxy_i * corr(Proxy_i, T_ins)) (it isn't, but its like that. Avg() is over i, corr is over the period of the instrumental record, the proxy series is assumed to run over a longer period). So if you reverse the sign of Proxy_i, it makes no difference at all to T_glob”
Given your articulation of the calculation, you are correct only if the corr(Proxy_i, T_ins) is moreorless constant through the full time period covered by Proxy_i.
[WMC #1]
Since Tiljander clearly indicates that the true relationship is swamped and in fact reversed by local non-temperature related activities in the recent period then (a) Mann should have excluded this proxy and (b) using it produces as spurious and as misleading a relationship as using the annual consumption of firewood in Finland over the last 500 years. It is simply a gross error and Mann (and you) should simply acknowledge it.
[WMC #2]
WMC inline responses
#1 - No, you've misunderstood. That corr is a single number, and is thus constant by definition. Were you under the impression that it was some kind of time-dependent running correlation?
#2 - If the proxy represents non-climate information then it shouldn't be used. But I've already said that, so I'm not sure why you want me to say it again. But now I have - are you hap-hap-happy?
AMac 12/7/09: According to Tiljander, there is no “if”--climate information in the Lake Korttajarvi data series is overwhelmed by non-climate influences in the late eighteenth, nineteenth, and twentieth centuries. This makes calibration to the instrumental record impossible. Thus, Mann used these proxies incorrectly (including the one or two of the four that was not upside-down).
Comment 30 | Posted by: Rattus Norvegicus | October 29, 2009 1:36 PM
A quick question.
AMac, you assert that in Finland colder temps lead to greater snowpack, however warmer air tends to hold more moisture and result in greater precipitation. For example, here in Montana we get our heaviest snowfalls when temps are just below freezing. When temps lower into the teens, even in the midst of a storm, snowfall amounts are typically less and the water content of said snow is nil. Since we live in a rather dry continental area this relationship might not hold true in Finland, but given what I know about snowfall and snowpack in Montana I would hazard that warmer winters might lead to greater snowpack and a larger snowmelt. There are also some confounding factors associated with late spring (or runoff season) temps. A good hot streak can cause rapid snowmelt which really loads up the rivers around here with sediment and shortens the runoff season. More normal temps during snowmelt extend the runoff season and the rivers have a lower sediment load. Of course these observations are strictly local in nature, but around here my guess would be that warmer temps would be indicated by thicker lake bed sediment deposition and colder temps by less deposition. Are you sure that you are correct about the way that Tiljander interpreted the thickness proxy and not the non-organic/organic matter proxies?
Comment 31 | Posted by: jeff id | October 29, 2009 1:41 PM
#25, You have to give McIntyre some room in his reply to PNAS the study had many flaws and everything had to fit inside 250 words.
Mann realized that the proxy was used upside down. I understand bender's comment but it seems simple enough to me that when you invert the scale on your thermometer it ain't temp anymore. Explaining the reason for the inversion was due to local environmental factors (which were pointed out by the Tiljander authors) and careless use of algorithms is another issue entirely.
Flipping an alleged temperature curve upside down and averaging with other alleged temperature curves is an obvious mistake...
#29, In the CPS method the orientation of the data does make a difference. In the EIV method it does not. Therefore, it does make a difference in the final result of the CPS method for certain, however we would need to see the resulting weight factors of EIV to determine if it was inverted there.
Now with all that said, EIV is trying to find the best fit. Since we have to invert tiljander to achieve an upslope and positive correlation it's highly likely that EIV also created the same problem.
Comment 32 | Posted by: AMac | October 29, 2009 2:05 PM
Rattus Norvegicus wrote (1:36pm) –
> [AMac], Are you sure that you are correct about the way that Tiljander interpreted the thickness proxy and not the non-organic/organic matter proxies?
I'm certainly not sure that I'm correct about Tiljander's interpretation! In fact, I have wondered the same thing. I'm a lay person (who was a long-ago geology major). I've never even been to Scandanavia.
However, I paraphrased Tiljander's interpretation of the data, which McIntyre reports has been supported by other Finnish scientists working in this area, and now by Kaufman.
It seems to me that the burden falls on Mann et al to explicitly state and defend a contrarian interpretation, if they are going to base reconstructions on it.
Read Mann et al's SI Methods treatment of the possible post-1720 unreliability of the varve record. They don't raise this issue, and imply (in my opinion) that they follow the standard interpretation.
I find it very unlikely that Mann et al realized that their algorithm imposed the opposite interpretation on the varve data. If they had, I think they would have dropped the varve proxies.
Comment 34 Posted by: Sean Houlihane | October 29, 2009 5:17 PM
I find it very interesting that the discussion is focusing on the sign of the proxy which is being discussed. It seems bizarre that this is viewed as the substantive issue when it does appear fairly clear that the proxy is not suitable for use due to contamination of the modern period - a fact which seems to be quite plain once the sign and coefficient of the proxy's contribution to the reconstruction are known (or guessed).
Is this a game of semantic point scoring, or is a discussion of the validity of this proxy in this specific method relevant here?
WMC response: This post, as the title rather suggests, was written in response to RP's confusion. The main aspect of that appeared to be his misunderstanding of the “upside down” issue. The issue of whether this particular proxy is of any use or not is another matter, doubtless fascinating in itself
AMac 12/7/09: There are no indications that Pielke was confused as far as the upside-down usage by Mann et al of two of the Lake Korttajarvi proxies. Pielke’s understanding was correct.
Comment 38 Posted by: PaulD | October 30, 2009 3:30 PM
“Is this a game of semantic point scoring, or is a discussion of the validity of this proxy in this specific method relevant here?”
This comment strikes me as dead on. The essence of SM's point is that Mann's algorithm misused the proxy by orienting it during the reconstruction period in a manner opposite of the physical theory that justifies its use as a temperature proxy. That is what SM meant by saying it was used “upside down” and I think that he certainly makes this clear on his blog, but perhaps less so in his published comment with its space limits. I also understand that “multivariate regression methods are insensitive to the sign of predictors.” The statement is true and entirely beside the point of SM's criticism.
I've read Pielke's post and it is clear to me that he understands SM's argument. He may have been confused by the statement, “Multivariate regression methods are insensitive to the sign of predictors.” I can also understand why his “confusion” arose, because I was similarly confused at first. He likely thought, as I did intitially, that the statement was responsive to SM's criticism, whereas it is actually a dodge, perhaps made intentionally or quite possibly based on a misinterpretation of SM's brief published comment.
The bottom line, however, is that I think it is unfair to criticize Pielke when he accurately understood the major point of the controversy, although he may have been a little confused on a technical statistical issue that is ultimately irrelevant.
WMC response: You're welcome to your opinion, of course, but I see nothing in Pielke's post that suggests he has a clue. What makes you think he “accurately understood the major point of the controversy”? As for "may have been a little confused on a technical statistical issue" I think you're being very generous.
AMac 12/7/09: PaulD’s characterization is succinct and correct, in my opinion.
--- End Abridged Stoat Comment Thread ---
Summary: Pro-AGW-Consensus scorecard
My view of the position of WMC (and the other technically-savvy pro-AGW-Consensus bloggers and commenters who contributed to the Oh dear oh dear oh dear oh dear thread) on the way that Mann et al (PNAS, 2008) used the Lake Korttajarvi varve proxies.
| Issue | Problem Acknowledged? | Call for Correction? |
| 1. Upside-Down Proxies | No | No |
| 2. Wrongly-Calibrated Proxies | Yes (equivocally) | No |
| 3. Other Major Problems with Temperature Reconstructions | To be determined | No |
AMac,
ReplyDeleteI've scanned your comments about Tiljander, so, as briefly as I can...
I'm unimpressed with the canned 'response' that multiple regression is insensitive to the sign of input series. Criticisms obviously relate to the signs on regression coefficients running contrary to a priori physical understanding. The effect probably ought to have been screened (yeah, 1000+ proxies makes that a lot of work, life is tough). I would bet on McIntyre being right, though I wouldn't bet much money. More generally, I've come to dislike the 'throw it all in and see what comes out of the soup'-type correlational analysis. OTOH Mann et al. did cross-validate and ran an impressive number of sensitivity tests.
The alleged error has little effect on results and conclusions. I don't understand why you think that it is
lethal to Mann et al (PNAS, 2008)
All models, all data, and all methodologies are flawed. Do the flaws matter? This comes back to effect sizes.
'Does Mann et al. contain correlations which are physically implausible but do not substantially effect the results?', is not an interesting question from a science point of view. What are the physical signals contained in the Tiljander proxies?, is an interesting question. Why do they correlate with temperature?, is another.
Pro-AGW-Consensus argument that a data series cannot be used upside-down
I don't understand how that is meant to be 'pro-consensus'. 'The consensus' is a list of broadly similar, widely shared beliefs that are held by climate scientists based on their interpretation of the scientific literature. I cannot think of any consensus statement being significantly effected by the use of Tiljander proxies in one (or two) studies.
it strikes me that leaders of the AGW Consensus have an overarching
There are no leaders. The consensus is not a social group.
If the varve XRD proxy is inconsequential, why include it at all?
That was not known a priori. The corollary is, if it is inconsequential, why remove it?
The publication [...] ought to be) extremely embarrassing to the high-impact PNAS and its editors and peer reviewers
Peer review is not expected to check signs on regression coefficients, and generally doesn't.
Those who defend the indefensible--because of whatever mix of ignorance, team loyalty, hubris, and confirmation bias--forfeit the presumption of trust on related issues.
That's not just Nick Stokes (Lucia's, yesterday) and Gavin Schmidt. It's many others as well (including Bob Grumbine, who commented earlier on this thread). Nobody is dissenting from the "party line."
Nick Stokes states;
Mann [...] came out with a proxy that Tiljander reasonably argues is the wrong way around [...] I would discard it.
Which are opinions I share. We both agree with the scientific consensus. What more would you desire/expect from us?
Robert Grumbine was taking apart, in his usual educational style, a boneheaded claim by Watts, as well as urging caution with regard to other claims being made. I have no prolem with either.
So, I'm not really sure what or whom "the party line" is meant to represent. Are some being overly defensive of Mann et al.? Probably. Are others greatly overhyping the importance of alleged issues? Again, probably.
Lazar, thanks for the comments. A couple of responses.
ReplyDeleteI meant "lethal to Mann et al" to mean lethal to the paper in its current form. The uses of the Tiljander proxies are wrong. Mann can certainly correct the Methods and Results. He'd have to do that before we could see what that revised paper looked like.
I do not agree that "The alleged error has little effect on results and conclusions." First, the version of Figure 8a (from memory) that you linked to is not the original that is in the SI at the PNAS website (last I checked). He's changed it twice since publication. Altering stuff without properly documenting it: this is poor practice that has the effect of hobbling critics.
Second, it's very difficult to understand what these figures show.
How many proxies were used to calculate each step? At each step, which proxies ended up being heavily weighted? Does each of the, say, 25 proxies contribute 4% to the signal? Or are a small number crucial, with the rest effectively serving as bystanders?
Consider the case of the (NH CPS and NH CPS minus-7) figure that you link. Let's engage our knowledge that all four of the Tiljander proxies were wildly miscalibrated, and that two (probably three) of the four Tiljander proxies are also inverted. The Tiljander series thus contribute a great deal of bad information.
Yet the CPS minus 7 trace (green line) is virtually superimposable over the original CPS trace (black line) with the exception of two spots between 1600 and 1720. In the as-published version, the superimposability is almost complete.
Taking out very-bad information and ending up with virtually the same reconstruction. Two ways to accomplish this: (1) If you gave the bad information almost no weight in the first place, or (2) If confirmation bias has led you to construct an algorithm that returns more-or-less the same hockey stick trace, over a wide range of inputs.
Who has looked at at the inputs and outputs of this paper and claims to understand what's going on here? For the major figures, how many proxies go into the curve for each century's output? How are they weighted? Would the substitution of "red noise" pseudoproxies for the actual proxies yield a trendless curve, or just another hockey stick?
-- continues --
-- continuing --
ReplyDeleteA simple and to me likely explanation is Confirmation Bias. Mann et al set out to identify a novel set of proxies to prove that the previously-studied tree-ring proxies told the right paleoclimate story: the hockey stick story. They identified candidates, qualified them, ran the analysis, found the expected pattern, and published.
But "hockey stick" might not be the correct story. There are other outcomes to consider.
Examples:
a. MWP temperatures might have been as high as today's, and proper analysis of relevant proxies would show that.
b. Available proxies may be so noisy that a meaningful temperature history cannot be teased out of them.
A null hypothesis would be, "there was no temperature trend, 0 AD - 1850 AD." So: if one takes a set of synthetically-generated 2,000 year pseudoproxies and runs them through the algorithm, does it return this no-trend result? The "red noise" criticism of Roman M and Jean S (Climate Audit) and Jeff Id (the Air Vent) is that Mann's procedure will generate hockey stick graphs from such input. I have not done such analyses myself (I'm far from fluent in R and other languages). Given the track record of these three, I give a lot of weight to their criticism. I'm not aware that Mann has addressed the point.
When I got into looking at Mann et al (2008), it didn't occur to me that people would defend the obvious mistake of retaining uncalibratable upside-down proxies. With these crude points not out of contention, I haven't returned to any of the subtler arguments for some time (till just now, at your prompting).
Nick Stokes offers, "I would discard it [the upside-down proxies]." In his Response to McIntosh and McKitrick, Mann does not.
Gavin Schmidt goes further, ending his analysis of Mann et al (2008) with "No correction needed." Nick Stokes endorses Gavin's remark.
So these discussions have a circular quality. X is right, well you proved that X is wrong, if I were Mann I would fix X, Mann should fix X, come to think of it his figure claims X doesn't make a difference, Mann refused to fix X, thus No Correction Needed, in other words X is fine as is, Mann need not fix X.
Let's see (1) the authors acknowledge their mistakes on Tiljander instead of scorning critics for noticing their errors ("bizarre"). Then let's see (2) the authors correct their results so that they aren't contaminated by the obvious errors. At that point we can (3) go on to constructively discuss the merits and shortcomings of the repaired paper.
That would be preferable to this Tiljander Proxy remake of Groundhog Day.
Lazar, thanks again for writing out your thoughts. Sorry I didn't get to all of the points you raised.
AMac,
ReplyDeleteMy time is severely strained so this may be my last response. As my friend TCO (*) might say, it would be better if you disaggregated these issues; the stuff you write on procedural issues, e.g. notification and documentation of changes, from; new methodological issues you raise, from; the actual effects on the given reconstruction of including/excluding Tiljander proxies. These are seperate issues.
I do not agree that "The alleged error has little effect on results and conclusions." First, the version of Figure 8a (from memory) that you linked to is not the original
This (the fact of a change to the published materials) is very far from my meaning of effect on results and conclusions. Results are the output of the algorithm, which are little changed after excluding upside-down Tiljander proxies. The conclusions include;
We find that the hemispheric-scale warmth of the past decade for the NH is likely anomalous in the context of not just the past 1,000 years, as suggested in previous work, but longer. This conclusion appears to hold for at least the past 1,300 years
... the 1000-year conclusion remains unchanged. You may argue that the 1300-year conclusion needs to be revised, if you also exclude tree-ring proxies. Maybe. So the inclusion of an updated graph shows to me... that the conclusions are (relatively) unchanged.
that is in the SI at the PNAS website (last I checked). He's changed it twice since publication. Altering stuff without properly documenting it: this is poor practice that has the effect of hobbling critics.
This may be true or not, but it is a seperate issue. The following are also seperate issues;
How many proxies were used to calculate each step? [...] Does each of the, say, 25 proxies contribute 4% to the signal? Or are a small number crucial, with the rest effectively serving as bystanders? [...] A simple and to me likely explanation is Confirmation Bias. Mann et al set out to identify a novel set of proxies to prove that the previously-studied tree-ring proxies told the right paleoclimate story
This last is in my view highly speculative, I do not find evidence for the claim.
--continued--
Another new issue;
ReplyDeleteThe "red noise" criticism of Roman M and Jean S (Climate Audit) and Jeff Id (the Air Vent) is that Mann's procedure will generate hockey stick graphs from such input.
I don't know the precise nature of the criticisms. So I don't know if what I say will be of any use or relevance. A relatively high likelihood of obtaining a hockey-stick shape from red noise is unavoidable if you're correlating with the instrumental record. The trend in the target series is matched whilst everything outside tends to flatline. So such a criticism (as I've presented it) is true, but it is also somewhat facile. Cross-validation is important. You can generate a hockey-stick shape with red noise, sure. Can you get it to pass validation? Note that the 'blade' is not flat in Mann et al., there are peaks and troughs where we expect them to be from energy-balance models and other studies.
Bauer, E., M. Claussen, V. Brovkin, and A. Huenerbein (2003), Assessing climate forcings of the Earth system for the past millennium, Geophys. Res. Lett, 30(6), 1276, doi:10.1029/2002GL016639.
Oerlemans, J. (2005), Extracting a climate signal from 169 glacier records, Science, 308(5722), 675, doi:10.1126/science.1107046.
So these discussions have a circular quality. X is right, well you proved that X is wrong, if I were Mann I would fix X, Mann should fix X, come to think of it his figure claims X doesn't make a difference, Mann refused to fix X, thus No Correction Needed, in other words X is fine as is, Mann need not fix X.
I think, from an scientist's 'I want to understand nature' point of view, it is sufficient to demonstrate that an alleged error can only have trivial effects on the output of an algorithm. Acknowledging the nature of the allegations is important again but is another issue. How notification is issued is another (procedural) issue again. I think all these issues are being mixed up in the various responses.
Anyway, thanks for engaging.
* If you're into McIntyre's criticisms of Mann's various output, it's worthwhile reading TCO's comments at CA.
Lazar, that's a great comment, and thanks for the tip on TCO's remarks.
ReplyDeleteThinking on your prior writing helped me to clarify what I see as a central problem with Tiljander. I know it's annoying to be faced with addressing a sprawling critique. So here's a focused one. What with time constraints, food for thought at least for now.
I think that the Tiljander proxies represent one thing in Mann '08, as the authors saw it. However, once the authors acknowledge the errors in their use of these proxies, it becomes possible to see that the authors' assumptions were wrong. Tiljander's proxies are, instead, important in a very different way.
The frame of reference of Mann '08's authors was:
"We have a good system for extracting temperature signals out of proxies to generate paleoclimate reconstructions. Let's go well beyond bristlecones and search for other proxies. In the case of the Tiljander proxies: are they valid? Do the concerns of the original authors disqualify them? Well -- we'll do the analysis with and without Tiljander (Supplemental Figure 8a). The reconstruction with Tiljander are nearly identical or very similar to those without (*). We conclude that the Tiljander proxies tell a story that is consistent with those told by the other proxies we've employed."
(*) Eyeballing the different Supplemental Figure 8a's:
Original SF8a, "NH CPS" (green line) is virtually identical with "NH CPS minus 7" (black line).
Once-revised SF8a, "NH CPS" (green line) is virtually identical with "NH CPS minus 7" (black line), except 1020-1100, 1120-1200, 1350-1390, and 1630-1640.
Twice-revised SF8a, "NH CPS" (black line) is virtually identical with "NH CPS minus 7" (green line), except 1610 and 1640-1680.
--- continues ---
--- continuing ---
ReplyDeleteWith hindsight, readers have important information that these authors lacked in 2008. We know that the Tiljander proxies are about the worst possible data to add into a temperature reconstruction. Since they are uncalibratable, all four are grossly miscalibrated to the instrumental record. The "skill" test that was done by splitting the 1850-1995 period in half gave encouraging results that we now know were entirely spurious. Worse, two and probably three of the proxies were added in to the reconstruction algorithim in an upside-down orientation. Whatever "warmer" information they contain was misread as "colder", and vice versa.
So, with this insight, we can see that SF8a is best seen as a test of a null hypothesis.
Null Hypothesis: "The combination of Mann 08's proxy selection protocol and computational algorithm does not produce a valid temperature reconstruction."
"We have derived a paleotemperature reconstruction that we believe has merit ("NH CPS minus 7"). If we pollute the input data with spurious information to generate a pseudo-reconstruction ("NH CPS"), the pseudo-reconstruction will deviate greatly from the genuine one. If it does not, that demonstrates that the algorithm we used to construct the genuine reconstruction is refractory to corrupt, meaningless input. In that case, we must accept the null hypothesis."
Alas, the null hypothesis should be accepted.
--- continues ---
--- concluding ---
ReplyDeleteThere are doubtlessly ways for Mann et al to argue this point differently from how I've presented it. By all means, they should do so. When they submit their corrected version of the paper as a belated response to McIntyre and McKitrick's 2009 Comment.
But the extended Groundhog Day arguments about "No correction needed" on the upside-down use of the Tiljander proxies serve to excuse Mann et al from facing up to their responsibilities as scientists. The Methods should be correct, certainly as regards errors of the magnitude of using uncalibratable proxies upside-down. The Discussion should squarely discuss the data that Mann et al generated and presented.
They should not be using PNAS as a vanity press.