In my first writeup, I observed that Exponent’s Logo transients appeared to be bodged too high, even with their unwarranted and adverse use of 67 deg F initialization (Exponent’s “temperature trick”). In today’s post, I’ve taken a closer look at the seemingly questionable calculation of the transients at 67 deg F, showing that the Patriot transients make sense only if initialization for the transients purporting to show Logo Gauge initialization were not actually initialized at 12.5 psi using the Logo Gauge (as stated and as is the purpose of the diagram). My reverse engineering shows that the Patriot dry transient in Figure 27 only makes sense if the Logo Gauge read 12.81 psi at initialization or if the Master Gauge (not the stated Logo Gauge) was erroneously used for initialization. If I’m correct, this is a very significant error – a botch, rather than a bodge – for which one would expect a prompt corrigendum, if not retraction, of the corresponding calculations. In a postscript to today’s post, I’ve attached a note on conversion from Logo and Non-Logo Gauge scale to correctly calibrated Master Gauge scale.
Figure 27 – Simulations with Logo Initialization
The critical simulations pertain to initialization with the Logo gauge. These simulations yield the transients in Figure 26, 27 and 30, that are used for comparison of “models” and observations. As previously noted, Exponent carried out these simulations at 67 deg F – the temperature most disadvantageous to Patriots. Their reasons for this adverse assumption were poorly supported and, in my opinion, are completely invalid. I’ve also observed a puzzling discrepancy between the transients shown in these figures and calculations using the Ideal Gas Law.
First, here is an annotated version of Wells Report Figure 27, showing dry (solid) and wet (dashed) transients for Colt (blue) and Patriot (red) footballs, each supposedly initialized at 13.0 and 12.5 psig at 67 deg F using the Logo gauge. In each case, I’ve shown my reverse-engineered estimates for the actual pressure reading (Logo Gauge) at initialization in order to yield the reported dry transients: 12.81 psig (Patriot) and 13.26 psig (Colt) – 0.31 and 0.26 psi respectively above the Logo readings said to have been used. In each case, I’ve shown the location of transients displaced downwards by these amounts. I’ve also shown (solid dots) the Logo average half-time (converted to Master scale using Exponent’s formulas), each coinciding with the horizontal lines in the diagram. Below the diagram, I’ll explain these calculations and speculate on how the discrepancy may have occurred.
Figure 1. Annotation of Wells Report Figure 27 as explained in text.
The solid dots showing the average Logo half-time measurements converted to Master scale (both shown at plausible times( confirm that the conversion formulas have been accurately implemented.
I digitized the Colt dry transient (solid steelblue) and fit it with a negative exponential to an asymptote. The corresponding y-intercept is 11.90 psig (Master). The pre-game pressure (Master) at 67 deg F initialization required to yield this pressure at 48 deg F ambient is 12.90 psig (Master), which, in turn, is equivalent to 13.26 psig (Logo) using Exponent’s conversion formula. At the stated initialization pressure of 13.0 psig (Logo) at 67 deg F, the corresponding y-intercept is 11.67 psig, about 0.23 psi lower than the y-intercept in Figure 27. A downward translation of the dry transient by this difference of 0.23 psi yields the 13.0 psig transient shown in thicker blue (together with wet transient a further 0.45 psi lower in dashed blue overlapping the Patriot dry transient).
Similarly, the y-intercept of the digitized Patriot dry transient in Figure 27 is ~11.43 psig. The pre-game pressure (Master) at 67 deg F required to yield this pressure at 48 deg F is 12.47 psig (Master), equivalent to 12.81 psig in Logo scale according to Exponent’s conversion formula, ~0.31 psi greater than the 12.5 psig (Non-Logo) said to have been used. A downward translation of the Patriot dry transient by this difference of 0.31 psi yields the 12.5 psig transient shown in thicker red (together with wet transient a further 0.45 psi lower in dashed red.) Note that the Figure 27 Patriot transient is consistent with it being calculated at 12.5 psig initialization using the Master Gauge but not with 12.5 psig initialization using the stated Logo Gauge. Similarly, the Figure 27 Colt transient is consistent with 12.9 psig initialization using the Master Gauge, but not with 13.0 initialization using the Logo Gauge. This seems like a gross error on Exponent’s part. I’ve attached a detailed script documenting my calculation. Using 71 deg F initialization, all transients move lower still.
Figure 25 – Simulations with Non-Logo Initialization
For comparison, the corresponding calculations for the Non-Logo simulations (Figure 25) are analysed identically below. In this case, the transients in Figure 25 are consistent with Colt calculations based on 13.1 psig (Non-Logo; 13.05 Master), as opposed to the stated 13.0 psig, and Patriot calculations based on 12.66 psig (Non-Logo; 12.61 Master), as opposed to the stated 12.5 psig, with re-calculate transients translated downwards by a corresponding amount as shown below.
Figure 2. Annotation of Wells Report Figure 25 (Non-Logo) as explained in text.
The wet differentials for both Colts and Patriots in this figure (~0.2 psi) are considerably less than the wet differential shown in the Logo diagram (~0.45 psi). For reference, wet differentials (dashed lines) using this larger differential are also shown in this figure. As in Figure 1, the average Non-Logo half-time measurements converted to Master scale are shown (solid dots) at plausible times.
Discussion
Exponent clearly stated that its Figure 27 simulations were initialized with the Logo Gauge rather than the Master Gauge:
In recognition of the remaining uncertainty as to which gauge was used to measure the footballs pre-game and in the interest of completeness, similar tests were run using the Logo Gauge. The Logo Gauge was used to set the pressure of two balls to 12.50 psig (representative of the Patriots) and two balls to 13.00 psig (representative of the Colts). From each set (corresponding to each team), one ball remained dry while exposed to the game temperature and the other was wet.
The above Figure 1 shows that this simply isn’t possible. The correctness of this observation can be confirmed by re-calling that initialization at 67 deg F and 12.5 psig yielded half-time pressures at 48 deg F ambient of ~11.5 psig, a value mentioned on several occasions in the Wells Report and observable in Table 10. The Patriot dry transient in Figure 27 is consistent with initialization using the Master Gauge, but not with initialization using the Logo Gauge, as stated and as supposedly the point of the comparison.
This seems like a botch, rather than a bodge, on Exponent’s part and, if so, ought to require a corrigendum, if not retraction.
These differences of 0.3 psi may not seem like much, but the amount in dispute is only ~0.3 psi they are highly material. Also note that any lowering of the transients typically increases the time window in which observations are consistent with transients. Using 71 deg F initialization, all transients move lower still.
Postscript: Conversion to Master Gauge
Before comparing “models” to observations, there are some thorny instrumentation problems that need to be addressed first. Not only was the Logo Gauge biased relative to the Non-Logo Gauge, but it experienced very large drift during Exponent’s studies. Reading between the lines of the report, it looks like they didn’t think about intra-study drift until after it had occurred, very much complicating and somewhat, in my opinion, compromising their conversion of gauge readings to correct (Master Gauge) pressures.
As a preamble, the gap between Logo and Non-Logo gauge measurements at half-time and post-game averaged ~0.38 psi, with no discernible trend between 10.5 and 13.25 psig – see the black +-signs in the figure at right below (overplotted onto an excerpt from Wells Report Figure 12.) This ~0.38 psi Logo bias can be seen in Figure 11 of the Wells Report (see excerpt in left panel below) as the bias at 13 psi in the first (Initial – lightblue) tests; the corresponding test for the Non-Logo Gauge showed no bias. The left panel diagram shows that the Logo Gauge bias had increased to ~0.75 psi by the time of the Final test, with the Non-Logo Gauge now biased low by ~0.15 psi.
Figure 12 of the Wells Report (excerpt in right panel) shows two calibrations (V1, V2) over the Logo and Non-Logo gauges over pressure ranges. Both the V1 and V2 calibrations show considerably larger differences between Logo and Non-Logo readings than actually observed on Game Day, and thus even the V1 tests appear to already experience drift from Game Day readings.
Exponent presented formulas for conversion from Logo and Non-Logo scales to Master scale, with the formulas said to have been calculated from an “early” set of measurements. The differences from Master scale arising from these formulas is shown as the dotted black (Logo) and dotted green (Non-Logo) lines in the right panel. The gap between the lines given by the formulas is noticeably less than the consistent ~0.38 psi observed gap on Game Day, strongly suggesting that some drift had already taken place by the time that Exponent calculated its calibration formulas. The failure of their conversion formulas to preserve this observed difference is a real count against this aspect of their technical work and a frustration to comparative analysis.
Figure 3. Left – excerpt from Wells Report Figure 11, showing Logo and Non-Logo gauge readings in successive tests: note drift upwards of Logo Gauge. Right – excerpt from Wells Report Figure 12. See text for further explanation.
Update: August 5, 2015
he error discussed in this post was also discussed by Back Picks here: http://www.backpicks.com/2015/07/25/deflategate-exponents-bias-and-the-master-error/. They illustrated Exponent’s error:
41 Comments
This sort or egregious error by Exponent leaves one gaping with astonishment.
According to Wikipedia, Ted Wells is a partner in Paul, Weiss, Rifkind, Wharton and Garrison, LLP., also known as Paul, Weiss.
This firm is HQ in NYC and operates eight offices nationally, with ~ 850 attorneys, total. Gross revenue, 2014: $934.5 million.
They will be in close touch with Exponent concerning these posts by Steve McIntyre, it can be assumed.
If they do not consult an independent source concerning Steve’s findings here, but rely exclusively on Exponent for determining the questions raised, it will be another marvel in this most interesting affair.
Esponent’s history involves some high profile simulations, including the Challenger explosion.
Even if Paul, Weiss were aware of these criticisms – and, even if they are, they could argue that they aren’t- they would presumably argue that they relied on qualified technical consultants and are not obliged to take account of everything on the internet.
Having said that, it does appear that this particular error is a misrepresentation of the methodology that they used and that they cannot get the result using the stated methodology. Reminds me in a way of Gergis et al.
I don’t think that you can assume that Paul, Weiss will necessarily do anything.
This cause is extremely “high profile”, in the parlance of the legal profession and Paul,Weiss has something at stake, and perhaps more than its reputation. If they’re not concerned about the questions raised (here and elsewhere) about the reliability of the Exponent study, it would be another marvel, imo.
I have just read that Ted Wells publicly defended (via a conference call)his report and the integrity of his investigation on May 12, I think. One Reisner, another attorney there was quoted on the thoroughness and reliability of the study by Exponent.
OOPS
Lots of reports on the web on this.
I’ve got a question. The form of analysis in this post is leading very quickly to an interesting Bayesian style analysis. To get there, I need some sort of prior – informative or uninformative. I’m wondering what sort of target deflation (distribution) could be presented as an alternative. If Patriots had a deflation plan, would they try to deflate by 2 psi, 1.5 psi, 1 psi, 0.5 psi, 0.1 psi? Has anyone seen any discussion of how much deflation might have been a target on other occasions?
Are you also planning to look at the, possibly related, issue of the Patriots’ unmatched success at avoiding fumbles?
Steve: since this issue wasn’t raised in the Wells Report, no. My interest in the issue is mainly how Exponent and Wells Report handled the matter, rather than investigating all possible theories on the matter – an approach that is comparable to my approach to climate issues.
If the topic interests you, I noticed a couple of discussions linked below. In the first article http://www.backpicks.com/2015/05/17/fumbling-statistics-and-patriot-trends/, the author observes that during 2007-2014, the Patriots had the least fumbles in the league, with Atlanta second. The difference was 3 fumbles over 7 seasons, which doesn’t seem like a large margin, though you may disagree. In the second article http://www.backpicks.com/2015/05/30/do-the-patriots-fumble-less-at-home/, the author observed that Patriots fumble rate away was almost identical to at home: indeed, the Patriots were the most uniform of all teams. It doesn’t surprise me that the most successful team over the period has the least fumbles, any more than the best tennis players making the fewest errors.
The fact is that the Patriots have been very successful under Belichek It would be most interesting to try to determine exactly what he does differently Instead, it seems that those who were beaten are utterly desperate to come up with excuses. I have never been a Patriot fan, but would love to see the Redskins try to adopt some of Belichek’s unique methods, whatever they may be.
An article in the Washington Post today by Sally Jenkins http://www.washingtonpost.com/sports/redskins/roger-goodell-has-one-credible-option-pardon-brady-blame-lax-protocols/2015/06/29/b2c12e28-1e76-11e5-bf41-c23f5d3face1_story.html was scathing toward NFL Commissioner Roger Goodell. In effect, it said that Goodell had no option but to undo the mess that he has made.
No doubt Jenkins has learned of the posts here by Steve.
The article was preceded by one written a few days ago, likewise critical of Goodell in this Deflategate affair.
Jenkins claims that Wells (Paul,Weiss), billed $5 million for the worked performed in the 243 page report (legal fees in NYC run upwards of $1,000/hr).
She gave no source for that very high figure.
Which brings us to another very interesting question:
How much money does the NFL owe for an egregiously flawed report? Obviously Goodell meant for the Patriot organization to make a substantial contribution to that fee, via the $ million fine.
Now it looks as Goodell might have to ask Ted Wells for an “adjustment” on the bill. All so very interesting.
Unless Wells or Exponent retract some or all of their report, Goodell is in a very difficult situation because any perception that he is going easy on Brady will be viewed with racial overtones – a point emphasized by, for example, Pardon the Interruption’s Mike Wilbon. If Wells and/or Exponent were trying to be objective, then they should issue a corrigendum or retraction of much of their technical work. It wouldn’t be pleasant but, at some point, it is surely their obligation if their report is supposed to be independent, as opposed to a prosecution brief. If it is merely a prosecution brief, then the NFL should not have been making decisions based on it alone in the first place.
Then Goodell is “impaled on the horns of a dilemma”? Poor fellow.
I sent the following email to Ted Wells:
I wonder if your avoidance of the phrase ‘more likely “than not”‘ at a crucial juncture in your letter was diplomatically intended. It could certainly have been used a time or two…
Steve,
Did you receive any response to your email to Ted Wells? If so, are you able to share the response and any follow-up communications?
Steve: no acknowledgement.
Oh, and check out this little nugget, a quote from a judge’s ruling against the Exponent regarding the disclosure of materials pursuant to their scientific methodology in another case:
http://thornography.weei.com/sports/boston/2015/06/29/wells-report-science-firm-exponent-gets-whacked-by-court-order/
“[T]he Court cannot allow Exponent to stand in violation of a valid Court order compelling the production of documents which were demanded pursuant to a lawful subpoena and found relevant by the Court,” Judge Stephen A. Stobbs wrote in the June 2, 2015 order. “Methodologically sound science has nothing to fear from full and open disclosure.”
Hmm… now what other field of science have I heard this kind of back and forth before…
“If it is merely a prosecution brief, then the NFL should not have been making decisions based on it alone in the first place.”
If you were presenting to a jury, that statement would go at the end of the summation, amirite? You need a bit of poetry here, like “if the transient doesn’t fit, you must acquit.”*
*for the British readers and those below the age of 40, see O.J. Simpson murder trial.
Isn’t the transient something they measured in simulations? How can they calculate a transient?
Steve: it was measured in simulations. But the simulations are subject to the Ideal Gas Law and ought to be accurately estimated for dry footballs. Their transient is inconsistent at the stated initialization.
A widening gap in readings between the two gauges can be seen in the record for the Patriots balls as they were measured at half-time as well.
Regarding the fumbles, well, Deadspin did something pointing out problems with the initial analysis. Brady ranks among the fastest quarterbacks in getting the ball out and on punt returns and kickoffs each team uses the same ball. It’s the little things that screw up such a great story:
http://regressing.deadspin.com/why-those-statistics-about-the-patriots-fumbles-are-mos-1681805710
Steve,
John C Hearn
Chief Professional Responsibility Officer, Paul, Weiss
Note his office. Perhaps a copy to him would be appropriate:
jhearn@paulweiss.com
Steve: good idea. Will do tomorrow.
Salamano:
Very interesting link. The judge lowered the boom on Exponent. Fancy that they imagined that they could defy a court order!
The court case in question appears to be an asbestos liability case. The lawyers are asbestos plaintiff litigators and Exponent has acted for asbestos defendants. “JM Company” is presumably a descendant of Johns Manville. Googling Exponent and asbestos litigation, turns up many interesting items, including schemes to keep scientific data confidential under attorney-client privilege while at the same time publishing the results in academic journals to provide citeability.
See http://www.publicintegrity.org/2013/10/21/13559/facing-lawsuits-over-deadly-asbestos-paper-giant-launched-secretive-research
http://annhyg.oxfordjournals.org/content/57/8/963.full.pdf+html
I wonder if Exponent and Wells have been involved in the NFL concussion case.
Brad Karl is given as the Paul,Weiss partner who represented the NFL in the concussion case. I could find no references to Exponent in that case. Wikipedia gives some interesting information on Exponent.
From Wikipedia, “Until 1998, it was known as Failure Analysis Associates and its holding company, The Failure Group, Inc., was traded as the symbol FAIL.” Seems appropriate. 😉
In the asbestos cases, Exponent appears to have been involved in publishing scientific studies that supported the industry’s cases. Exponent also has a scientist specializing in concussions http://www.exponent.com/william_bussone/
I haven’t looked at his work, but I wouldn’t be surprised if the conclusions were favorable to the NFL, if one were to extrapolate from Exponent’s asbestos techniques.
BTW I just watched a local sports talk show in which Luke Willson, tight end for Seattle, was interviewed at length. Willson is originally from southern Ontario. Willson was an extremely engaging young man. They asked him about concussions. He said that Seattle taught styles of tackling without using the head and he thought that the dangers had already been much reduced. Maybe just the optimism of a young guy. He said he was a big believer in playing multiple sports and he was playing squash in the offseason. I immediately paid more attention to him.
Oh my…just a cursory look on the web shows all sorts of cases this firm has done with the NFL. Also some NHL too. The refs I believe are direct employees of the NFL—and are chosen by the NFL, not teams or individual owners. There is an utter conflict here in favor of the refs, or in finding their activities not inadequate or negligent, as extensions of the NFL itself.
Here is info from Attorney Reisner’s bio:
Successfully represented the National Hockey League in various litigation matters, including defending use of the “Minnesota Wild” trademark by the NHL’s Minnesota franchise … Successfully represented the National Football League in various litigation matters, including winning an injunction against an ambush marketing campaign that infringed NFL trademarks.
This defiance of a court order intrigues me. It is impossible to believe that they have nothing to hide.
#####
Speedy et al vs
3M Corp. et al
in the Third Judicial District of Illinois
Plaintiff, age 25, diagnosed with mesothelioma; claims exposure to asbestos as a child.
Co-defendents include Boeing, United Technologies,Georgia Pacific, Honeywell International, Goodyear Tire and Rubber, others.
Exponent claimed privilege for certain documents not so classified under Illinois law. Seems to bear on Exponents methodologies in asbestos related studies. The court on May 11 ordered Exponent to deliver the documents to the Plaintiffs. Exponent has defied this order.
From the ruling of June 2, 2015:
“Exponent has failed to comply with this court’s order and has indicted that it will continue to disregard this court’s May 11 order”
So Exponent thinks to appeal. They must have sound reasons for wanting to hide their methodologies. I am reminded of Steve’s FOI fight against Keith Briffa and the CRU.
Rethinking this affair, it seems likely that Wells would amend the report to read, from “more probable than not” to ” uncertain”, if he got the nod from Goodell, the client. The “prosecution” would simply be dropped.
There is no question that Goodell needs to re-evaluate his position;he is looking at a lawsuit from Brady supported by the Players Association, and all hinges on dubious science by dubious Exponent.
I would guess that Goodell will find away to get clear of this mess.
Except for the fact that he p—-d away a reported $5 million of the league’s money.
Have you considered sending this to Daniel R. Marlow – Professor of Physics
Princeton University? He supposedly advised Wells on the science and seems respected by his peers. He may be someone who is concerned about his credibility – cause I don’t think either Exponent nor Wells do. marlow@princeton.edu
If he advised Wells, he did so for a fee and he thus has a particular point of view in this matter, perhaps.
Great work steve. Another site has analyzed fumble rates, brady’s performance home vs. road as well as the science and other parts of the wells report etc. here: http://www.backpicks.com/
Methinks that I perceive another botch in Exponent’s transient curves. I refer to the wet ball transient curves (perhaps this has been raised before). This error can be best seen in the Wells Report fig. 21 (fig 1 in Steve’s analysis).
Note that the wet ball transient at halftime shows wet ball re-pressurization at the same rate as the dry balls for the first few minutes. This is impossible, if I’m not mistaken. I do not see how wet balls can begin to warm/re-pressurize until they have become dry balls.
If, for example, it took two minutes for a damp ball to dry, then warming/repressurization should not begin until that point. From that point onward,the transient becomes a dry ball transient. The wet ball transient in Wells fig 21 is incomprehensible when viewed in light of this consideration.
Shifting the warming/re-pressurization of wet balls by two minutes into the half-time interval will have the effect of shifting measured psi down by 2.5-3 psi, it appears.
Correction,
by 0.25-0.3 psi
http://www.backpicks.com/2015/07/25/deflategate-exponents-bias-and-the-master-error/
I have just posted a comment at this blog concerning the game day drop in barometric pressure as per Norwood Airport: 7 MB (0.1 psi) from 4 pm to 8 pm, approximately. This should have caused the same in all of the game balls.
If you have not sent to the Patriots, they would certainly have interest. Stacey James, Vice President of Media Relations, staceyj@patriots.com, 508-326-6868 would be a good start
The error discussed in this post was also discussed by Back Picks here: http://www.backpicks.com/2015/07/25/deflategate-exponents-bias-and-the-master-error/. They illustrated Exponent’s error:

Steve, why don’t you email Mike Florio and talk to him about all of the exponent errors?
Nicely written article http://business.financialpost.com/fp-comment/deflating-deflategate-confusion-about-basic-science-could-have-wrongly-smeared-tom-brady
Robert Blecker filed and amicus with the Appeals Court
Click to access Deflategate-Amicus.pdf
3 Trackbacks
[…] an earlier article, I pointed out that essential figures in the Exponent report contained (what appeared to be) an […]
[…] game temperature and the other was wet. For reasons set out in more detail at Climate Audit (https://climateaudit.org/2015/06/29/exponents-transients-bodge-or-botch/), it is impossible that the bolded statement is a correct description of Exponent’s actual […]
[…] contemporary evidence of such a possibility. A news story shortly after the game (on NFL.com here) reports an interview with D’Qwell Jackson, a Colt cornerback who intercepted a Brady pass in […]