In a May 2018 “Policy Position” for the John Locke Foundation, Kathleen Restrepo wrote the following:
A study conducted by University of North Carolina and North Carolina State University researchers found that patients seeking treatment from Access Healthcare, a direct-care practice located in Apex, North Carolina, spent 85 percent less on total health care spending and enjoyed an average of 35 minutes per visit compared to eight minutes in a nondirect-care practice setting.
https://www.johnlocke.org/policy-position/direct-primary-care/
Can you imagine that?
Did Restrepo imagine it?
Let’s carefully address her sourcing and find out.
Restrepo misrepresented the provenance of the 85% claim.
If you thought that Restrepo’s hyperlink from the word “study” to an article in a peer-reviewed academic journal would take you to an academic report of the study by a team of academic research professionals, you were wrong. Restrepo’s statement is not your ordinary reference to a piece of peer-reviewed academic research.
Restrepo gives a fourth-hand account of unpublished material by medical students and business school students engaged in course work projects. The published article by Eskew and Klink (neither of whom was a researcher from a North Carolina university) , to which Restropo provided a rather misleading link, gives the third-hand account of the research Restrepo describes; the second-hand account of that particular research comprised less than three minutes and three powerpoint slides in a meeting presentation by Dr. Brian Forrest.
The business school students’ actual work was never compiled into a manuscript, although the students made slides and presented them in several closed-to-the public venues (personal communication with Charles Queen, one of three business student authors named by Forrest ). Forrest’s talk also included a thirty second summary of separate work by an unstated number of unidentified medical students.
Along with the identity of the originators of any work referred to, the very fact of publication and the details of publication are, of course, important initial indicators of the credibility of cited research. Even high-school students are taught to fully and accurately represent the provenance of the material they reference. Restrepo knew that the relevant work was entirely by students (see her earlier policy piece), but eschewed revealing that telling detail to those she sought to influence. More importantly, even though the Eskew and Klink article to which Restrepo pointed plainly stated that the actual research was unpublished, Restrepo disguised that unpublished research by dressing it in the garb of a peer-reviewed published article.
Restrepo did not accurately convey the content of the article she cited; and that article had not accurately conveyed the content of the source it cited.
High-school students are also taught that they must accurately represent, not just the provenance of claims on which they rely, but also the substance of the material to which they refer. Yet it seems that Restrepo’s fourth-hand account may have failed even to accurately convey what was said in Eskew and Klink’s third-hand account. Eskew and Klink (“EK”) say the study showed that DPC patients “spend 85% less out of pocket for their total cost of care compared with the same level and amount of care in a traditional setting.” Restrepo offers instead that DPC patients “spend 85% less on total health care spending”. These seem to mean quite very different things. Dr. Eskew has confirmed to me that he was referring to primary care cost sharing for insured FFS patients. But primary care costs are only a part of “total health care spending”, referred to by Restrepo.
Perhaps Procrustes could fit Eskew, Klink, and Restrepo on the same page. If so, shouldn’t that page be the Brian Forrest presentation that Eskew and Klink had identified as their source? But neither Restrepo’s fourth-hand account nor Eskew and Klink’s third-hand account accurately reflects Forrest’s second hand account of what the research team itself had to say about comparative savings cost savings for DPC versus traditional patients.
The 33rd minute of the Brian Forest talk relied on by Eskew and Klink was the only point at which Dr. Forrest referred to comparative cost savings of DPC versus traditional patients as determined by NCSU business students. For this, he showed a slide by those students which made exactly one cost comparsion: that of the employee share of premium for various employer sponsored insurance policies versus the full premium of a catastrophic policy ; the students computed a differential of 33%.
On the other hand, the 18th minute of his talk was the only point at which Forrest referred to any specific work by UNC medical students. There was no slide, but he said this, and this alone: “In fact, some work by some UNC medical students showed that people who were commercially insured actually came out of pocket 7% cheaper for the year when they came to our practice versus ten other local practices that were in the traditional model that were in network.” I have repeatedly asked Dr. Forrest for copies of any reports made these students or that he identify them; he has not answered.
Neither a 7% difference in OOP nor a 33% difference in insurance premiums bears much resemblance to the 85% reductions in whatever it was Eskew, Klink, and/or Restrepo (EKR) had written about. No 85% figure was tied to any student research finding anywhere in Forrest’s presentation. Somehow, the entire EKR trio found themselves in contradiction to the very report that announced the existence of the studies to which they referred! There’s no written explanation of the 85% number anywhere. And it is not even present in the student team’s slide presentation to a DPC conference.
Nothing could better demonstrate why it is broadly agreed that referrers should carefully examine the material to which they refer. This is precisely why the rules of citation prioritize primary reports of research results. Indeed, even when citation of secondary reports is allowed because, for example, the original source reference was physically unavailable for inspection, these rules nonetheless require details of the original source.
In short, the value of sharing research by citation turns on accuracy in describing both the provenance and the content of the material cited.
The 85% claim was masqueraded as high quality research precisely because it is literally incredible.
Eskew and Klink’s 2015 article in the Journal of the American Board of Family Medicine declared that unpublished work by post-baccalaureate students who studied a certain direct primary care clinic in 2013 “demonstrated” that the average fee for clinic members was 85% less than the cost-sharing paid by traditionally insured patients for the equivalent care. The 85% claim is preposterous.
The American Academy of Family Practice and affiliated groups regularly lament that 8% or less of health care costs are spent on primary care, and hold up 12 or 13% as an aspirational model. In 2013, the overwhelming majority of traditionally insured patients were covered by employer sponsored plans. These plans had an average premium of $5884 for a single adult and an actuarial value of about 87.5%, indicating average total health care costs of about $6725. Even if we apply AAFP’s aspirational 13%, the amount spent for primary care by insurers and insureds combined would be less $875. Reducing that by 85%, would mean that the direct primary care practice in question was receiving fees of less than $132 per person per year. That’s not credible.
As the NCSU students showed, however, the average member of the subject DPC practice paid fees of $473 per year. But, in that case, 85% savings would imply that primary care spending in traditional FFS practices was $3,153, about 47% of total health care costs. That’s AAFP’s aspiration more than tripled. That’s not credible, either.
And then there is Katherine Restrepo, who gilded the 85% lily by assigning that huge reduction to total health care costs, not merely primary care costs. That would mean that the DPC patients had per annum costs for all health care of $1009 dollars. Subtracting the $473 they pay for primary care, that leaves $536 dollars for all downstream care. But for average FFS insured, even the aspirational 13% allocation for primary care leaves 87% for downstream care – $5851. Dividing $536 for downstream care of those DPC patients by $5851 for downstream care for FFS patients suggests that the Apex DPC’s patients saw a truly miraculous 91% reduction in downstream care costs. Nowhere near credible.
In a separate post, I explain that Restrepo’s suggestion that DPC office visits can be over four times as long as traditional office visits, is equally incredible. For now, keep in mind that Restrepo apparently expects the public to believe that DPC both has vastly lower costs and delivers vastly longer visits.
If you are a doctor choosing treatment for your sister, feel free to rely on third-hand and fourth-hand reports of literally incredible results of unpublished medical research by Master’s level students, some unnamed. If, instead, you are treating my sister, make sure you’ve paid your malpractice premium.
Please approach the design of healthcare systems that serve our brothers and sisters across the country with some concern for credible evidence.
DPC Referee 