undefined | Better HN

0 pointsthaumasiotes4y ago0 comments

> Its lunacy for tests to be regulated beyond requiring rough false positive false negative rates

There is a good reason tests are described in terms of sensitivity and specificity ("if the answer in reality is yes, how often will the test say no?") rather than in terms of false positive or false negative rates ("if the test says no, how often is the true answer yes?"). The sensitivity and specificity are facts about the test which can conceptually stay constant[1] as you apply the test to different people. False positive and false negative rates do not have that property; they are facts about the group you're performing tests on just as much as they're facts about the test.

[1] This is not to say that the sensitivity and specificity of a test do stay constant as you apply the test to different populations. Often they won't. But it is a theoretical possibility, and even that isn't true for false negative rates.

0 comments

midjji4y ago

For a test manufacturer you are right, and they aren't responsible for accounting for general population frequencies, or conditional frequencies based on things like some sibling has it etc. But there is no need to regulate that. But the serivice in question isnt a test manufacturer, its diagnostics, and they are responsible for this. I think a company selling a diagnostics service should be required to prominently display reasonably accurate FP,TP,FN,TN rates for gen pop, as the next example will show.

This is is not a legal problem, its a customer experience one, and as such its very much what the free market does well. A clever company could say that the tests service we provide works in a two stage process where a preliminary test is used to identify which conditions need to be investigated further, the results of which are presented in a meeting with one of our doctors were the results are explained and additional tests are performed to resolve any ambiguity. This combines the benefits of sensitivity, and specificity in a resource efficient way, and does not provide the user with any scary results before a certain answer is available. The FP,FN,TP,TN probabilities for gen pop for such a company would be much better than those of companies trying to do so in a single pass. But for these figures to mean anything regulating that they are available and reasonably accurate is key to ensure an efficient market.

Another problem is of course that the tests themselves are shit, or the rates based on poor data. The requirement on reasonably accurately reported gen pop rates help with this however, and I see no way around legally forcing diagnostic services providers to prove this. And until someone does, my patent pending algorithm(){return false} tests for all rare genetic disorders with excellent accuracy^^

OJFord4y ago

I'm not sure GP was intending to make that distinction (rates globally for the test vs. the sample population being tested by.. what a given doctor/hospital?) - I haven't come across that before.

If the population is the same then your changed-order definitions are just inverses, and they're just different terms for the same thing.

thaumasiotesOP4y ago

> I'm not sure GP was intending to make that distinction (rates globally for the test vs. the sample population being tested by.. what a given doctor/hospital?) - I haven't come across that before.

> If the population is the same then your changed-order definitions are just inverses

No, you just haven't understood the concept.

Let's assume some condition has a prevalence of 20%, and a test for it will correctly identify presence of the condition 95% of the time, while correctly identifying absence of the condition 90% of the time. We can immediately answer the first question: when the answer is "yes", the test will say "no" 5% of the time.

You have proposed that when the test says "no", the answer is "yes" a share of the time that might be the inverse of 5%, or perhaps 5% itself. I have no idea what you meant -- and I suspect you didn't either -- but the correct rate of false negatives is not 5%, 95%, nor 2,000%.

In a model population of 10,000 people, we will see this:

                  |  condition present  |  absent  |
    test positive |               1900  |     800  |
         negative |                100  |    7200  |

From this table we can see that the false negative rate is 100/7300 or 1.4%. The false negative rate looks much better than the sensitivity and specificity figures because the condition is rare. The corollary to that is a horrific false positive rate of 800/2700 = 30%.

OJFord4y ago

I don't think I have, it sounds like you think I'm saying false negatives are the inverse of false positives? Not at all, that's obviously not true.

I was surprised at 'sensitivity and specificity' (jointly) being considered different from 'false negatives and false positives' (jointly).

The given reasoning was about population differences, which.. fair enough, I understand that makes a difference, I just wasn't aware that was a standard difference in definition (if it is) and suggested the up thread commenter wasn't (or wasn't meaning to use it) either.

> correctly identify presence of the condition 95% of the time, while correctly identifying absence of the condition 90% of the time. We can immediately answer the first question: when the answer is "yes", the test will say "no" 5% of the time. You have proposed that when the test says "no", the answer is "yes" a share of the time that might be the inverse of 5%, or perhaps 5% itself. I have no idea what you meant -- and I suspect you didn't either

10%. 'inverse', as I called it, of 90%, not 95%.

(That's why I think you think I think (..!) that false negatives/positives rates are derivable from one another. Sorry if not and I'm just still not getting it...)

I don't think I am misunderstanding though - Wikipedia calls them 'true pos/neg rate', and gives formulae for false pos/neg rates as 1-true: https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity

1 more reply

j / k navigate · click thread line to collapse

0 comments

midjji4y ago

OJFord4y ago

I'm not sure GP was intending to make that distinction (rates globally for the test vs. the sample population being tested by.. what a given doctor/hospital?) - I haven't come across that before.

If the population is the same then your changed-order definitions are just inverses, and they're just different terms for the same thing.

thaumasiotesOP4y ago

> I'm not sure GP was intending to make that distinction (rates globally for the test vs. the sample population being tested by.. what a given doctor/hospital?) - I haven't come across that before.

> If the population is the same then your changed-order definitions are just inverses

No, you just haven't understood the concept.

In a model population of 10,000 people, we will see this:

                  |  condition present  |  absent  |
    test positive |               1900  |     800  |
         negative |                100  |    7200  |

OJFord4y ago

I don't think I have, it sounds like you think I'm saying false negatives are the inverse of false positives? Not at all, that's obviously not true.

I was surprised at 'sensitivity and specificity' (jointly) being considered different from 'false negatives and false positives' (jointly).

10%. 'inverse', as I called it, of 90%, not 95%.

(That's why I think you think I think (..!) that false negatives/positives rates are derivable from one another. Sorry if not and I'm just still not getting it...)

1 more reply

j / k navigate · click thread line to collapse