It was absolute hell. The key problem here is the waiting and uncertainty. You have the NIPT at 10w, but you can’t have the amniocentesis until several weeks later. When that came back fine, there were questions about whether it was a “mosaic” meaning only a small proportion of cells are effected. We were only really in the clear after the 20 week ultrasound.
That’s a lot of weeks to be consumed by wondering about whether to terminate the pregnancy, or wait it out for more information. I have a masters in bioinformatics (in genomics!) and my knowledge of stats and the science was next to useless in the face of these decisions.
I know of couples who simply couldn’t deal with this uncertainty and chose to terminate on the basis of this test alone.
Fortunately for us our child was fine and is a perfectly healthy 18 month old now, but I wouldn’t do the rare trisomy test again.
The risks associated with this extended testing are just not worth it, perhaps aside from Down's (from a numbers point of view). Even then, there are many completely gorgeous children and people with Down's .. chances are you'll have a curveball in life one way or another at any rate.
I have two children, one diagnosed with ADHD/ASD, the other likely not too different but too early to tell. Apple doesn't fall too far from the tree. Wouldn't change a thing, other than to avoid the ABA services companies like plague - they prey on your insecurities and you might face financial ruin for possibly no real benefit to the child if you go along with their spiels.
So, to see medical companies exploiting vulnerable new parents who will do anything for their children? I am shocked. (/S...)
The tragedy of receiving news like this is probably fathomable, but I think it may be hard to grasp the emotional and intellectual agony of deciding whether to terminate a pregnancy based on a set of probabilities.
It breaks my heart to think that parents face this decision with erroneous data.
The point is that people who get a "positive" result on these tests are often put through terrifying levels of anxiety when there is no actual problem; this anxiety is often exacerbated because they aren't informed of the false positive rate. This clearly has a harmful emotional effect on people, and explaining the false positives in Bayesian terms, or reframing it in terms of sensitivity and specificity, doesn't undo that damage.
That potential harm needs to be explained to patients, and it needs to be weighed carefully against the potential benefits of the test (as is done for PSA tests for prostate cancer, which also have a high false positive rate). Given that potential for harm, it's not unreasonable to ask that these tests be more tightly regulated.
To quote the OP:
> In interviews, 14 patients who got false positives said the experience was agonizing. They recalled frantically researching conditions they’d never heard of, followed by sleepless nights and days hiding their bulging bellies from friends. Eight said they never received any information about the possibility of a false positive, and five recalled that their doctor treated the test results as definitive.
(Edit: clarified)
No you should not panic, anymore than you should celebrate and buy a yatch if you think you have 10% chance of winning a lottery.
The doctor should phrase it as "the test indicates you have a 10% possibility of cancer. The majority of people who test positive do not have cancer. Further testing is required to confirm or rule it out."
Going with the expected value of 10% of a horrid thing is still bad, or 10% of $100m is still $10m, is not applicable to a single non repeated event.
There is a good reason tests are described in terms of sensitivity and specificity ("if the answer in reality is yes, how often will the test say no?") rather than in terms of false positive or false negative rates ("if the test says no, how often is the true answer yes?"). The sensitivity and specificity are facts about the test which can conceptually stay constant[1] as you apply the test to different people. False positive and false negative rates do not have that property; they are facts about the group you're performing tests on just as much as they're facts about the test.
[1] This is not to say that the sensitivity and specificity of a test do stay constant as you apply the test to different populations. Often they won't. But it is a theoretical possibility, and even that isn't true for false negative rates.
This is is not a legal problem, its a customer experience one, and as such its very much what the free market does well. A clever company could say that the tests service we provide works in a two stage process where a preliminary test is used to identify which conditions need to be investigated further, the results of which are presented in a meeting with one of our doctors were the results are explained and additional tests are performed to resolve any ambiguity. This combines the benefits of sensitivity, and specificity in a resource efficient way, and does not provide the user with any scary results before a certain answer is available. The FP,FN,TP,TN probabilities for gen pop for such a company would be much better than those of companies trying to do so in a single pass. But for these figures to mean anything regulating that they are available and reasonably accurate is key to ensure an efficient market.
Another problem is of course that the tests themselves are shit, or the rates based on poor data. The requirement on reasonably accurately reported gen pop rates help with this however, and I see no way around legally forcing diagnostic services providers to prove this. And until someone does, my patent pending algorithm(){return false} tests for all rare genetic disorders with excellent accuracy^^
If the population is the same then your changed-order definitions are just inverses, and they're just different terms for the same thing.
The solution is fairly straightforward I think: require the test companies to report the posterior probability of the illness given the test result. Probably with some kind of uncertainty interval.
I don’t really understand what your implication is. Are you wishing you had not done the genetic tests? Suggesting you should have aborted the 20-week fetus based on unusual skull appearance? Suggesting that the meningitis was a result of some malpractice? Mad at medical professionals who are now “dropping hints” without saying something more substantive?
The problem with these tests highlighted (confusingly) by the NY Times article is that they test for extremely rare conditions but have a high enough false positive rate (by my napkin math, on order 0.05% = 1/2000 false positive rate) that they end up dramatically over-diagnosing these conditions.
That doesn’t necessarily mean the people designing the tests were lacking in “sound science” or were behaving maliciously. Their tests just need to be almost perfect to not overdiagnose rare abnormalities, and perfection in medical tests is a very high bar. Arguably we shouldn’t do tests for such rare conditions unless the false positive rate can be dropped by another order of magnitude or more, and if we do such tests communication to those who test positive should be much clearer. On the other hand, since genetic abnormalities are so life altering, even a 10% or 20% chance of an accurate positive might be grounds to abort a very early fetus.
It sounds like your specific problem was that the genetic tests did not return a positive result, even though it seems plausible that your child had some genetic abnormality. Not picking up every possible genetic condition is a different problem from the false-positive problem highlighted by the article.
Unlike a false negative where there isnt enough to trigger the test, false positive implies that something triggered a positive.
You have something, just not what the test thinks you have. For example, men with testicular cancer, apparently, show up as pregnant in pregnancy tests. Clearly thats a false positive for pregnancy.
84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course)
84% of people whose world is turned upside down are actually getting a wrong diagnosis.
This article is talking about precision, which is the proportion of positive results that are true. And it's okay for precision to be awful, especially when the condition is so rare. But it's only okay if the result is communicated alongside a statement saying what the precision is, which it seems these were not.
> the tests are inaccurate, when in reality the tests are accurate
If the test make someone consider terminating a pregnancy or even considering it, that's a lot of pain. So for that human, the test is failing its purpose potentially, depending on the value calculation of terminating a viable pregnancy vs the severity of the issue if it comes to term.
For a human, accuracy as you defined it means little to nothing. Usefulness and helpfulness are far better metrics, and such a high false positive rate is clearly causing issues in respect to those, which is what the article is highlighting.
It’s clear the article is talking about why sensitivity is important in layman’s terms and while it could use better writing it’s a real problem in diagnostics. This is why you don’t ask men to take a pregnancy test to check for prostrate cancer. It is accurate but not sensitive.
It mixes up poor communication (reporting false-positive/negative rates as if posterior probabilities, & exaggerated confidence thereof), arbitrary-seeming decision thresholds (but their hyperventilating over '85% wrong' notwithstanding, many are probably too conservative, if anything, given how devastating many of these problems are, there should be more false positives to trigger additional testing, not less), costs of testing (sure why not but little is presented), tests which they claim just bad and uninformative (developed based on far too little _n_, certainly possible), implicit calls for the FDA to Do Something and ban the tests (not an iota of cost-benefit considered nor any self-reflection about whether we want the FDA involved in anything at all these days)... Sometimes in the same paragraph.
Plenty of valid stuff could be written about each issue, but they'd have to be at least 4 different articles of equivalent length to shed more light than heat.
This is true in so many areas of journalism but lately seems especially egregious in the NYT. And I don't really blame them, as the incentives for any individual reporter are just too great - having the government make a major policy change based on your article is basically the brass ring for an investigative reporter.
I basically can only use these types of articles as a jumping off point for my own research, as I usually find the moralizing conclusion the article comes to as unsupported.
Yep, this is the framing I came here looking for.
Investigative journalists live in the same asymmetrically-incentivized world as social science researchers. If the reporter had looked into the phenomenon and concluded "yeah, boring technical logic pretty much works as expected here" then there's no story.
I'd like to imagine there's an investigative-journalism-editor-news somewhere, and they're discussing this discussion saying "bah, these clowns are making sweeping generalizations about editorial standards based on only a false positive; this is totally specious with no mention of the prior distribution or sensitivity vs specificity trade-offs"
Not that you're necessarily wrong, but how did you get that from the article? It didn't seem to me like they wanted a ban.
I doubt that hyping one side of this equation is fraud. Pushing the error in this direction seems like a good idea, anyway. If you have some weird illness, and the test comes back as a false positive, at least you'll continue to explore that possibility for a while. If it comes back as a false negative, then you'll spend a ton of time exploring alternatives which will be true negatives. Probably infuriating.
https://www.harmonytest.com/content/dam/RMS/harmonytest/glob...
https://web.archive.org/web/20211116203541/https://myriadwom...
https://images.health.questdiagnostics.com/Web/QuestDiagnost...
I'd appreciate it if you could point out where any of them walk the potential customer through sensitivity, specificity, and the fact that if they test positive, there is an 80-90% chance that their will not be affected. I can't seem to find any of that.
This mindset is ingrained in every doctor I speak to, but I think it's just so wrong.
Take DiGeorge syndrome. You have a 1/4000 chance of having it, and the test carries an 81% chance of a false positive. The above doctor calls this "marketing"? Foolishness. That's an incredibly useful test. The downside is small, and the upside is asymmetrically large.
We need far, far better screening for all sorts of things. Adult cancer and heart screens once a year, prenatal screening, and on. We do a good job with breast and prostate screens, but for rarer conditions our current approach of waiting for the disease to be symptomatic makes no sense. Part of that will be driving the cost down. There is so much market need for a legitimate version of Theranos and I'm glad there are some companies working on these things.
Do we? Unless I'm missing something, breast cancer is a huge killer and PSA tests are deeply imperfect. I am very much not expert in these areas.
You are supposed to treat a positive on NIPT as “there’s a chance your baby has this, need a more accurate procedure to confirm”.
It sounds like their ob gyn wasn’t able to explain results to them or they didn’t understand the probabilities. To be fair our provider didn’t even suggest tests for the disorders in the article, probably because of the false positive rates and rarity. Sounds like these extra screens shouldn’t be offered.
We have been undergoing IVF with my wife since 2019. (Covid made a huge mess of those plans...) One of our embryos tested as a possible positive (but only slightly) for aneuploidy of one chromosome.
The doctor, a veteran of IVF, looked at the results and said "my experience is that this is either a very small mosaic error, which tends to be utterly invisible in real life, or a computer artifact. I have never seen embryos with those borderline results develop any serious problems later. Things would be different if the aneuploidy signals were clear, but definitely do not discard this embryo".
This _does_, however, seem like a framing issue, more than a utility issue. If the tests are 100% accurate at detecting true positives, they're a great aid. But rather than framing the tests as a be-all, end-all source for information, why not frame them as "a test that suggests whether or not you should get other tests"? That simple wording change would save a great deal of added stress on someone starting or growing a family.
Having been on the receiving end of a false positive, I'd still do the test again for a hypothetical future pregnancy. Even though it was hell for a couple of days.
That got knocked down pretty quickly but wow a lot of folks picked up a big chunk of their lifetime radiation allowance because of that.
These tests seem to operate under a similar model, disregard the risks of unnecessary testing in return for information of limited utility that may cause material harm.
To evaluate the value of performing a diagnostic test as an intervention, you DO have to look at final actual patient outcomes at an appropriate end target which includes sending people unnecessarily down different treatment paths, including additional testing with additional risks. And most importantly is that, in fact, mere knowledge of diagnostic results has been PROVEN to cause harm in many scenarios.
Now... if a patient WANTS that test, I think it should be available. But whether or not it should be performed routinely without prompting is an appropriate question for regulatory bodies.
https://twitter.com/JohnFPfaff/status/1477382805583716353?t=...
'For a disease w a 1-in-20,000 risk, a test w a false positive rate of 1% and a false negative rate of 0%—an insanely accurate test—would identify 1 correct case and 200 false positives every time. Or would be wrong 99.5% of the time.
This isn’t “bad tests.” This is… baserates.'
*) My wife insists that it was a doctor, not an RN, but my brain won’t let me process that possibility.
They revised the tests so the first test would say Inconclusive rather than Positive, and ask them to repeat it. This saved some lives.
Maybe this a UX failure? Shouldn't the test designers present the results like this, even to doctors?
"A 2014 study found that 6 percent of patients who screened positive obtained an abortion without getting another test to confirm the result."
Maybe people aren't informed enough. It is my experience that some doctors tend to cut conversations short and some people are shy/insecure enough not to pry answers out of them.
In this case, that would be a tragedy, given that statistically 5 of those 6 aborted fetuses were healthy.
Edit: I found the following comment in the comment section of this article, which appears to address the same issue:
I am a physician with a PhD in Biomedical Informatics. Most patients who receive these tests do not see a maternal fetal medicine doctor or genetic counselor, and no one actually explains that the tests they are receiving are “screening” or “diagnostic.” Your opinion that this article does a disservice to patients reflects your unrealistic assumption that most of the doctors ordering these tests are actually communicating effectively with patients (or frankly, even understand the tests themselves). In my experience, they usually aren’t/don’t. Articles like this “fill the gap” on patient education when doctors are unable to explain math and risk (i.e., most of the time).
Your results are sent directly to your provider, so you can’t read the fine print yourself. And if you do get access to the results, the wording implies that a null result (not enough DNA collected) actually means you’re likely to have some disorder. In fact the wording here actually got worse in the three years between my two (healthy) births.
Ideally these companies should require genetic counseling before you take the test. Parents should understand that these tests are for screening purposes only, and that a definitive diagnosis can’t be gotten until 16-20 weeks. Unfortunately these companies have found a niche- parents wanting to know the sex and health of their children as soon as possible- and have no real reason to improve their practices.
Considering this as a UX challenge - imagine a grid of 10,000 dots (100x100).
Draw one box around the base rate - the rate at which you expect to find the problem in the population. If the base rate is 1%, then the box is 10x10 = 100 dots.
Then color in the dots for the test positive rate (not false positive, just all positive tests) False positives would be the colored dots outside the box.
Next to that, put strikes through the dots corresponding to your expected false negative rate.
That said, I do feel that pulling in abortions to the debate is specifically to trigger a set of readers. But to what aim? They have not established that the tests could be better. Just that when they say yes, they are still not perfect.
