Bogus QA Processes - sm

Posted: Feb 26, 2010

A little lesson here in statistics, probability and sampling that might surprise you, so stick with me like a mustard plaster. First, a couple of definitions: "Population" = all the things you want to measure, but can't measure directly, usually for practical reasons. The "population" of an MT's work in a month would consist of ALL the reports she transcribed. "Sample" = some subset of the population that you CAN measure in order to make a statistical inference about the population. So, for instance, we query a SAMPLE of the entire POPULATION of people who eat out at restaurants to find out if they'd like to have snails on the menu and make an inference about the whole population based on the results from the sample. Let's say that we talk to 100 eaters-outers and 46 of them want snails. It would be nice if we could say that "46% of the population want snails", wouldn't it? Well, it turns out that we can't. In fact, we know that it is highly UNLIKELY that the results of querying the entire population will be 46%, like our sample. We know that the population results might be 44%, or perhaps 48%, and so we express our sample results with something called a "confidence interval" to express the level of imprecision in our inference. It would look like this: 46% plus or minus 2%, for instance, to indicate that the best we can say from our sample is that we believe that 44 to 48% of the population wants to eat snails. But wait - we really need to go further and say HOW sure we are that the true population result DOES fall within this specified interval. Are we 90% sure? 95% sure? 99% sure? If I said "46% plus or minus 20%", or 26% to 66% of the population wants snails, I would obviously be much more sure that the population result falls somewhere in that range than if I say "46% plus or minus 1%", or 45% to 47% of the population wants snails. So if you've stuck with me, you will see that: 1. I can't derive a single number from a sample that I can say with confidence represents the result I would get from measuring the whole population. 2. I have to use a confidence interval (the "plus-minus" thingie) to express the range that I think would include the true population measurement. 3. The wider my confidence interval, the more certain I can be that it would include the true population measurement, and the narrower my confidence interval, the less certain I can be. So, if you do, say, 20 reports a day five days a week for four weeks, perhaps you do 400 reports in a month. I take, say, 20 of them for a sample and conclude that your "QA score is 98.3%". Dear me, there seems to be a flaw here, doesn't there? Something missing? Something that might be the CONFIDENCE INTERVAL? All I can REALLY say from my sample is that your QA score is 97.3% plus or minus some percentage, right? There are two ways we can "narrow" the confidence interval: 1. We can take a larger sample. Obviously, the closer we get to sampling the entire population, the less variable or smaller the degree of uncertainty we have about whether the range we're expressing includes the true population measurement. Small samples equal more uncertainty; large samples equal less uncertainty. 2. We can accept a lower confidence level. If I only have to be 90% sure that the confidence interval (the range) contains the true population value, I can use a smaller confidence interval than if I must be 99% sure. So - let's get back to our example. You've transcribed 400 reports in a month (we could use lines also, but this makes the math easier). I want to take a sample of your reports, and I want to be 95% certain that the results from that sample will be within 1% (plus or minus) of the "true" result I'd get from checking ALL your reports. How many of your 400 reports do you think I'd have to take as a sample? 20? 40? 60? ANSWER: 385. That's correct, sports fans. If I want to have 95% certainty that your "QA score" truly represents your performance (across all the work you did) within a range of plus-minus 1%, I will have to check nearly ALL of your 400 reports. So, let's ask the question another way: If I can only sample 40 reports from a population of 400, what will my 95% confidence interval be - in other words, what will be the range around the sample results that I'll have to use in order to be 95% sure that it would include your "true" population result? ANSWER: 14.7%. Your QA score would be, for instance, 97.3% plus-minus 14.7% However, these numbers assume something that probably isn't true, which is that the practical range of QA scores could run from 0 to 100%, and we would certainly hope this isn't so. Let's make a little correction, and say that the typical range of MT scores in our company runs from 95 to 100%. Now let's run our problems again: SAMPLE SIZE: 329 reports (out of 400, remember) for a confidence interval of plus-minus 1%. OR CONFIDENCE INTERVAL: Plus-minus 6.42% for a sample size of 40 reports. But wait - we're not done yet. ALL of the above assumes that the sample IS representative of the population, and it turns out that it is very difficult to come up with samples that aren't biased in one way or another. Many factors can introduce bias into samples - and especially small samples, which we have already seen have their own problems that are purely mathematical. Sample bias falls into two main categories: 1. Systematic bias: Our methods of sampling are flawed in such a way that the bias is "built in", which makes it somewhat predictable at least. Systematic bias is recognized when samples are always "off the mark" in one direction or the other (plus or minus) from the true population result. 2. Random bias: Our methods of sampling are flawed in such a way that accidental factors can skew the sample results in a significant way. There are methods we can use to deal with sample bias, but they are not without cost. First, we can go ahead and DO some full-population QA checks and compare the "true" results with a sample taken from the same population. Second, we can do some things that make the sampling process more representative of the population. For instance, we can recognize that certain types of reports are typically easier (e.g. ER notes versus operative reports, or ESL dictators versus native English dictators, etc.) and weight our sample to reflect the relative percentages of each type of report found in the entire population. So - if Suzie did 400 reports of which 70% were ER notes and 30% were op notes, we would not want a sample of 40 reports that contained 12 ER notes and 28 op notes. We would want 28 ER notes and 12 op notes instead (70% and 30%, respectively). This is called "stratified sampling", and obviously it requires attention to the mix of work that people do and attention to see that the sample is representative of that mix. Another thing we might want to do to protect the integrity of the sample would be to eliminate "outliers" from the sample. "Outliers" are jobs that are "atypical" for reasons beyond the MT's control. For instance...a dictation that had terrible sound quality compared to most. Or...a dictation that represented a worktype that this MT has never done before. (Yes, of course we want to QA these reports, but we do NOT make it part of her monthly QA!)... Are you still sticking with me? We will next address the completely arbitrary "point system" promoted by our old friends, AAMT, or whatever they're calling themselves these days!