Some thoughts about quality in transcription. - Warning - this post is not a sound bite.

A very short word on background.  In preparation for certification as a quality process analyst (American Society for Quality), I've reviewed AHDI's "Best Practices" recommendations, which has been at least tacitly accepted as a "standard" in the industry.  I chose to address this "standard" because (a) I have over two decades in the field in a variety of roles, and (b) I've known intuitively for some time that the standard was deeply flawed.  I felt it was time that the flaws should be exposed and articulated.  This is a brief excerpt from my analysis:

LISTED PARTICIPANTS:  I will reserve for my private judgment whether or not the participation of MTIA in creating this standard was appropriate, noting only that it creates at least the appearance of bias in the form of self-interest.  When it comes to creation of standards, absolute objectivity is everything.   On the other hand, the relationship between AHDI and the companies in this industry has raised more than one eyebrow anyway, so I'm not sure that inviting MTIA to stand down in this process would have really made much difference.

I found no one among the listed participants who are indicated as holding any of the various recognized certifications in the field of Quality Assurance.   I then looked for academic expertise.  The sole academician in the group had published some papers regarding peripheral issues such as "trust in distributed workgroups", etc., but I find no evidence of specific expertise in healthcare documentation, much less formal quality methods, prior to participating in this group.  Although he has published a couple of papers well AFTER participating in the group, it is difficult to see that he brought specific quality expertise to the group.  To his credit, he has written at least two very insightful papers about the transcription work process since that time, although that is not relevant to this question.

INITIAL IMPRESSION:  My first review of the AHDI quality "standard" left a distinct impression, i.e. that it consists of some cobbled-together boilerplate text derived from some books, that the statistical measures were applied incorrectly, and that no pathway from the protocols described to the goals expressed by the standard was articulated.  

FINAL IMPRESSION:  After a careful review of the standard, my impression is unchanged.  This is not a standard worthy of adoption by any transcription company, nor one that a transcription company's clients have any reason to rely upon.

SAMPLE OF SPECIFIC ISSUES:

1.  The assumed "fungibility" of errors:  By the logic expressed in the AHDI "standard", three noncritical errors comprise the same quality impact on a medical document as one critical error would have, merely by the simple process of addition.  It is possible to expose the fallacy of this logic by way of a simple example:

Let's suppose that in a typical document, the transcriptionist substitutes "discrete" for "discreet", transcribes "by 3:00 p.m." as "by 3 o'clock p.m.", and believes that she hears "in the" instead of "on the".

By application of the AHDI "simple addition" protocol, these three obviously inconsequential errors are the PRECISE EQUIVALENT of omitting the statement that the patient is allergic to penicillin.

Really?  Such outrageous outcomes are rampant because of the inexplicably unsophisticated approach that AHDI has taken to the "accumulation" of errors which, in fact, obfuscates the FUNDAMENTAL DISTINCTION between critical and noncritical errors.

2.  Although AHDI gives a brief nod to other sampling methods, the fact is that this "standard" advocates - and even describes - a particularly troublesome sampling method known as "random sampling".  Lest you, like AHDI, be seduced by the notion of "random sampling", BE AWARE that random sampling (a) is rarely truly random, and (b) is often inappropriate.

To be more specific, "random sampling" is surprisingly difficult to achieve.  More to the point, random samples obscure much of the most valuable information that sampling by other means can reveal.  THIS IS THE VERY INFORMATION NEEDED FROM QA PROGRAMS FOR THE ORGANIZATION TO MOVE FORWARD.

Again, a simple example:

Which is better: 

(a) To know that the MT has an overall score of 98.5%, or

(b) To realize that she achieves 99.4% on everything except cardiology procedure notes, where she achieves a score of 97.3%?

There are sampling methods - as easily executed as "random" methods - that will reveal this vital information.

3.  Every report starts off at 100 and is impacted equally by the "cumulative error count", regardless of length.  It shouldn't be necessary to describe the fallacy here.

4.  The statistical information provided by the "standard" is simply incorrect.  It is not possible to sample 1%, or 240 lines out of a population of 24,000 lines and achieve a confidence interval of 0.851 with 95% certainty. 

To explain sampling, confidence level and confidence interval, let's start with the fact that there are several questions that have to be answered when extrapolating from a sample to a population.  (By way of explanation, in statistical terms the "population" for MT quality review would be ALL of the lines that an MT transcribed during the period being reviewed.  A "sample" is some subset of that "population" - in other words, the reports selected for review.

We can't review every line, so we're going to take some sample of the population.  As described above, just how we select those reports matters, but let's move on.

Here are the questions that affect just what we can legitimately infer from a sample:

(a)  How sure do I need to be that the sample accurately reflects the population?  I can never be 100% sure, because even if only one member of the population isn't reviewed, it could be different from all the others.  99% is the highest degree of certainty achievable from a sample.  95% certainty is commonly accepted as a reasonable level.

So, I say that I want to be 95% certain that my sample represents all the work the MT has done during the period.

(b)  But this degree of certainty NEVER applies to a single number, such as "98.7%", unless there is no variability in the sample at all (e.g., every single report achieves a score of 98.7%).  Since there will be reports rated at 98.5%, 98.2%, 99.6%, etc., we can only be 95% sure of a range of values that represents the population.  This is called the confidence interval.

You've seen the confidence interval in political polls:  With 95% certainty, 48% of the population approves the tax measure, plus or minus 4%.  That "plus-or-minus" figure means that we are 95% sure that from 44 to 52% of the population approves the tax measure.  This is the confidence interval.

Now, here's the relationship between sample size, confidence level, and confidence interval:

*  For a given confidence level (say, 95%), if I want a smaller confidence interval (I want to be more precise in my analysis), I have to sample more of the population.

*  For a given confidence level, a smaller sample is going to give me a much wider confidence interval (I have to be less precise) than a larger sample.

In fact, a sample of 1% (240 sampled out of 24,000 total lines) as AHDI suggests, gives you a confidence interval that is almost 2.5%, not 0.851% as they describe. 

WHEN WAS THE LAST TIME YOU RECEIVED A QA SCORE WITH A CONFIDENCE INTERVAL ATTACHED TO IT?  Have you EVER gotten this:

"Dear MT Mook:  Your QA for the last period was 98.6%, plus-minus x%."

Not happening, is it?  You get a score of 98.6% as if it were just that simple, and just that accurate.

5.  The AHDI "standard" does not prescribe the obvious step that MUST be taken when a review falls below the acceptable level because of the unreliability of the AHDI protocol, and that is RESAMPLING - preferably using a larger sample.

But what happens instead?  Utter reliance is placed on the small sample, followed by nasty notes, recriminations, disciplinary actions, and sometimes termination.

6.  The AHDI "standard" prescribes no pathway to progress.  Why, for instance, is there no "sandbox" where MTs can learn without fear of failure?  In this environment, the MT could transcribe certain documents (e.g., certain dictators, certain worktypes) under "full review".  Documents coming out of the sandbox would never be included in a QA review, and all feedback would be of a training/educational nature.  Ideally, people would be paid by the hour rather than by the line while they're transcribing in the sandbox.  Getting it right would have precedence over production speed.

When I was a supervisor, I had plenty of time to run a sandbox.  The problem is, you see, that a sandbox does carry a "cost", and it's an investment that takes a while to see any return.  Hence, it's not an investment that MTSO's are willing to make.

7.  The AHDI standard articulates no metrics for measuring the QA process itself.  What are we getting for what we're doing?  How do we know what works, and what doesn't?

SUMMARY:  Notwithstanding the AHDI "standard", even in the 21st century the quality processes in medical transcription remain unnecessarily crude, are of questionable accuracy, and provide uncertain value to organization in terms of the stated objective of continuous improvement in quality.  The emphasis in QA remains one of  "sniffing out the guilty", and even the statistical means by which "the guilty" are identified are flawed.

We can do better than this!