In the latest match-up between human and machine translators, humans have once again come out on top.
IITA Translation Contest at Sejong Cyber University
The International Interpretation and Translations Association (IITA) hosted a contest at the Sejong Cyber University in Seoul, South Korea, pitting a team of four human translators against three Machine Translation (MT) programs, including Naver’s Papago, Google Translate, and Systran.
The Korea JoongAng Daily reports that the participants — both human and robot — were tasked with translating short, never-before-translated fiction and nonfiction texts from English to Korean and from Korean into English. Each human translator was given 50 minutes to translate one of the four passages included in the contest. Each MT program, on the other hand, translated all four passages — in a total of ten minutes.
However, while the MT programs certainly had the humans beat in the pace of their work, the quality of their translations fell far short of the standard set by their human competitors. A committee of professional translators led by Korean Association of Translators and Interpreters President Kwak Jung-Chul scored the human translators’ work at 49 out of 60 on a scale that combined measurements for “accuracy, language expression, and logic and organization,” according to the Daily. The highest-scoring MT program performed just slightly more than half as well as the human translators, earning a combined score of 28 out of 60.
What does the competition mean for the translation industry?
Now, while competitions like this are always entertaining to read about, in practice they don’t do much more than provide another opportunity for language tech developers and industry professionals to rehash the same argument we’ve been hearing since Google Translate officially launched in 2006. Developers insist MT is just a few years away from achieving parity with human translators, and translator and interpreter associations insist that, as good as MT programs might get, they’ll never achieve a totally ‘human’ level of sensitivity to the nuances of language.
The actual results of the competition pretty accurately reflect the terms of that debate. The Daily reports that the errors that resulted in the MT programs’ low score are of the sort that have traditionally marred translation programs’ work — grammatical errors stemming from the machines’ inability to understand what the texts’ authors were trying to say. The report includes one sample translation from Thomas L. Friedman’s book “Thank You for Being Late” — machines translated the English sentence “Steve reached into the top pocket of his jeans and pulled out the first iPhone” into a Korean text that read “Steve went into his jean’s pocket and pulled out the first iPhone.”
The difference between the original English sentence and the MT program’s translation is slight — “Steve reached into the top pocket of his jeans” vs. “Steve went into his jean’s pocket.” In fact, it’s so slight that the essential content of the sentence isn’t completely obscured — it’s still pretty clear that Steve is pulling the first iPhone out of his pocket. But the difference is large enough to interrupt any seamless reading of the text by drawing the reader’s attention away from the overall message and toward the text’s grammatical errors.
The devil is in the details
More importantly, the translation also omits some of the original sentence’s information (the bit about Steve reaching into the “top pocket” of his jeans). While that small change might not seem like an issue when recounting a story, it could become a serious problem in a legal or medical setting, where even small omissions can have an oversized impact on court proceedings or a patient’s course of treatment. For example, think what could happen if an MT program determined that it wasn’t essential to provide a verbatim translation of the part of a patient’s medical history that explained which of their kidneys was a transplant and which was their own.
As the Sejong Cyber University’s competition once again shows us, the MT programs currently on the market simply don’t have the sensitivity necessary to match up with the human translators who can catch those kinds of mistakes. The devil’s in the details, as the old adage goes, and that’s especially true when it comes to translation.