An efficient model for creating a parallel text corpus from a comparative text corpus

Master's Thesis in Computer Engineering (Software)

Abstract

Most of the modern translation approaches in the field of machine translation, including statistical machine translation, example-based machine translation, and combined machine translation, use a set of co-translated texts under the name of parallel text bodies as the main educational data. But for most languages, parallel bodies are available to a very small extent, or they are related to a specific range of texts. On the other side, there are adaptive bodies whose raw materials are easily obtained. Comparative corpuses do not include co-translation texts, but both texts in two different languages ??in terms of the similarity of criteria such as content, publication date, title, etc. are compatible with each other.

Compatible structures include sentences that can be a good translation for each other. The purpose of this treatise is to automatically construct a parallel corpus by extracting such sentences from the comparative corpus. The model presented in this research consists of three main steps: (1) selection of pairs of parallel candidate sentences using the filter of sentence length ratio and the filter of the number of common words (2) selection of pairs of parallel sentences using the maximum entropy classifier and considering the features related to the length of the two sentences, their common words and features based on the alignment at the word level between the two sentences (3) increasing the accuracy of the pairs of sentences extracted by selecting only one of the sentences paired with each sentence. This work can be done by calculating the proximity of that sentence by translating paired sentences from the other side by the TER criterion and selecting the closest sentence. Finally, the efficiency of the model presented in two parts (1) evaluation of the designed maximum entropy classifier and (2) evaluation of the usefulness of the extracted parallel sentence pairs in improving the quality of machine translation is examined. 

Chapter One

Introduction

Due to the increase in regional interconnections and the need for information exchange, the demand for language translation has greatly increased. Many documents need translation, including scientific and technical documents, manuals, legal documents, textbooks, advertising brochures, newspaper news, etc.; Some of them are difficult and challenging to translate, but most of them are boring and repetitive, and at the same time, they need coherence and precision. It is difficult for professional translators to meet the ever-increasing translation needs. In such a situation, machine translation can be used as an alternative.

After 65 years, machine translation is one of the oldest computer applications. Over the years, machine translation has been the focus of research by linguists, psychologists, philosophers, computer science scientists, and engineers. It is not an exaggeration to say that the new work in the field of machine translation has significantly contributed to the development of fields such as computational linguistics, artificial intelligence and program-oriented natural language processing.

(Images are available in the main file)

Machine translation can be defined as: "Translation from one natural language (source language) to another language (target language) using computerized systems and with or without human help". Research work in the field of machine translation is not limited to the grand goal of fully automated high quality (publishable) translation. Rough translations are often enough to revise foreign topics. Recent efforts are towards making limited applications in combination with speech recognition, especially for handheld devices. Machine translation can be used as a basis for further editing, translators usually use tools such as translation memories that use machine translation technology but put them under their control.

Machine translation is one of the research areas of "Computational Linguistics". So far, various methods have been invented to automate translation, which are categorized in different ways in the field of machine translation. Figure 1-1 shows the types of machine translation methods available in the form of categories as given in [1].

1-1-1. Dictionary-based machine translation[1]

This type of machine translation is based on dictionary entries; And in it, the word equivalent is used to produce the translation. The first generation of machine translation (from the late 1940s to the mid-1960s) was based entirely on electronic dictionaries. This method is still useful to some extent in translating phrases rather than sentences.. Most of the methods that were developed later use more or less bilingual dictionaries [1].

1-1-2. Rule-based machine translation[2]

(images are available in the main file)

Rule-based machine translation deals with morphological, syntactic and semantic information of source and target languages. Linguistic rules are built from this information. This method can deal with various linguistic phenomena and is extensible and maintainable, but the exceptions in the grammar add problems to this system. Also, its research process requires a lot of investment. The goal of rule-based machine translation is to convert source language structures into target language structures. This method has different approaches.

Direct approach [3]: source language words are translated without going through an intermediate representation. In this method, the context of the text, meaning and scope are not taken into account.

The transfer approach[4]: The transfer model belongs to the second generation of machine translation (from the mid-1960s to the 1980s). In this model, the source language is transferred to an abstraction that is a less language-specific representation. Then an equivalent representation for the target language (with the same level of abstraction) is produced using a bilingual dictionary and grammar rules.

Interlingual [5]: This method belongs to the third generation of machine translation. In this method, the source language is transformed into an intermediate language (display), which is independent of both participating languages ??(source and destination) in the translation. The translation for the target language is then derived from this auxiliary representation. Therefore, in this type of system, only two analysis and combination modules are needed. Also, due to the independence of this method from the source and destination languages, it is mostly used in multilingual translation machines. This method emphasizes a single representation of different languages.

1-1-3. Knowledge-based machine translation[6]

This method deals with a conceptual dictionary that represents a domain. This method includes two stages of analysis and production. The basic components of a knowledge-based translation machine are an ontology of concepts, a dictionary and grammar of the source language for the analysis process, a dictionary and grammar for the target language, and mapping rules between the syntax of the intermediate language and the source and target languages.

1-1-4. corpus-based machine translation[7]

The approach of corpus-based machine translation emerged in 1989 and was widely discussed in the field of machine translation; And because of the high accuracy of this method in translation, it was overcome by other methods. In this method, knowledge or translation model is automatically obtained from bilingual text corpus (set of texts). Since this approach works with large amounts of data, it has been called corpus-based machine translation. Some types of corpus-based methods are described below.

Statistical machine translation [8]

Although the initial idea of ??statistical machine translation was introduced by Warren Weaver in 1941, it has been widely used since 1993 when this method was modeled by IBM researchers; So that currently statistical machine translation is the most common approach in machine translation. In the statistical machine translation method, statistical models are used, and the parameters of these models are extracted from bilingual texts or "parallel bodies". In other words, the statistical machine translation system learns the translation probabilities from the parallel corpus and uses these probabilities to produce a suitable translation for the input sentences that have not been trained in the process. In this method, two main models are used, namely word-based models and phrase-based models.

Example-based machine translation[9]

Example-based machine translation methods are also called memory-based methods[10]. The idea of ??this method started in 1980 in Japan. These types of systems try to find a sentence similar to the input sentence in the parallel corpus, and then produce a translation of the input sentence by applying changes to the previously stored translated sentence.

The basic idea in this method is to use existing human translations to translate new texts. Therefore, it is enough to break the new texts into small pieces and search the equivalent translation of these pieces in a database of translated pieces and produce the desired translation. This method has data limitations. Compiling a very large set of examples also does not cover the entire language.