“Yandex” switched to a hybrid translation system using neural networks
Applications for text translation have made the life of many people easier, allowing them to communicate with residents of other countries and read information in different languages without the need to study them. But it’s no secret that the quality of such a translation often leaves much to be desired. Therefore, the company “Yandex” launched a corporate service “Yandex.Translator” hybrid translation system. A translation technology based on the neural network has been added to the statistical translator. Thus, the translation is performed at once by two systems, and then the algorithm based on the machine learning method CatBoost compares the results and offers the best.
The statistical translator translates each word and phrase individually, so the result is an unrelated text. Neural networks, on the other hand, process entire sentences. Due to this, the translated text is more precise and understandable. The statistical translator does not know how, but he remembers and translates rare and complex words and phrases well. Working together, both systems compensate for each other’s shortcomings.
Yandex employees note that there are many ways to train a machine from one language to another. For example, you can give her dictionaries, from which she will master the rules. You can also show her many parallel texts. Comparing them, the machine will learn to find matches – for example, so it will understand that the words “dog” and “dog” are likely translations of each other. This approach is based not on rules, but on statistics, where the name of this method came from.
Since the launch, Yandex.The translator has used only a statistical system. When translating, she divides sentences into separate parts and for each picks up all possible translations with an indication of their probability. Then the system makes up different variants of the new sentence from the translated fragments. The user is shown one that contains high-probability translations and in which the fragments fit well with each other.
Neural network translation
Like a statistical translator, a neural network analyzes an array of parallel texts and learns to find patterns in them. At the same time, the process of the translation itself is structured somewhat differently. A neural network does not work with words and phrases, but immediately with whole sentences. She receives an input in one language, and at the output, she gives the sentence in another language. This approach allows us to take into account the semantic links within the sentence. The neural network catches the essence of the sentence even in the case when the words that transmit it are in different parts of the sentence.
Neural network translation also has a number of shortcomings. If a neural network can not translate a sentence for any reason, it will simply begin to offer something most suitable, trying to guess the right answer. Also, neural network translation does not always work well with the translation of little-spread names, toponyms and other rare words.
The combination of the advantages of statistical neural network methods makes it possible to significantly improve the quality of the translation.