A community of 30,000 US Transcriptionist serving Medical Transcription Industry
If you've never heard yellow pronounced "jello" before, I'm sure it can totally throw you. Like learning the differences between L and R transpositions for Chinese and Japanese dictators (think Mulan = Chinese, so L is pronounceable but R becomes L, and Hiroshima = Japanese, so R is pronounceable but L becomes R ... very basically speaking).
And mentally changing the syllable being accented can sometimes help, for Middle Eastern and African ESLs especially. I spent way too long once trying to figure out an allergy that sounded like "fee-oh-REEN-al" (Fiorinal, arrgh).
I haven't had a lot of experience with Spanish-speaking (I'm guessing in this case?) dictators myself, but you've just learned one thing about them, haven't you? A few other tendencies you might notice:
http://answers.yahoo.com/question/index?qid=20101018165430AAsbtjC
http://www.speakingyourbest.com/accentguides/spanishaccent.html
The absolute worst for me are the Eastern European ones (those "nuclear wessels" types mentioned above). It's like I have a mental block that I can barely overcome in terms of reinterpretating a lot of what they're doing to English (beyond the obvious V = W), so I can certainly feel your pain.
That's a big factor too, just plain and simple exposure. I probably could do Spanish-speaking dictators fairly well too, for the same reason (having grown up in California), though probably not quite as easily as Asian ones, but I've just never much had the opportunity to find out. I'd trade a heavily accented Russian or Polish dictator for an Asian one any day.
And for what it's worth, in case anybody thinks my agreeing with the OP that some ESLs can be very difficult and frustrating makes me xenophobic, that's not at all the case, and I happen to know about and agree with the studies that show improved patient outcomes from IMGs (international medical graduates) compared to native-born graduates. I still think that the IMGs' outcomes and our jobs could be improved even more by higher English language requirements, though, frankly.
http://www.nytimes.com/2010/08/12/health/12chen.html?_r=0
http://www.kevinmd.com/blog/2010/08/international-medical-graduates-patient-outcomes.html