Title Page
Contents
Abstract 11
1. Introduction 13
2. Related Works 18
2.1. Ensemble in Machine Learning 18
2.2. Ensemble in Deep Learning 19
2.3. Ensemble in a Generative model 20
3. Methods 22
3.1. Baseline: Majority 23
3.2. Recurrent Ensemble with Survival 24
3.3. Recurrent Ensemble with Consensus 26
3.4. Case Study: Application of Recurrent Ensemble 27
3.4.1. Seq2Seq 27
3.4.2. Transformer 36
4. Experiments 46
4.1. Models 46
4.2. Datasets 48
4.3. Training Details 49
4.4. Performance Metrics 50
4.5. Results 52
4.5.1. Results in Seq2Seq 52
4.5.2. Results in Transformer 60
5. Conclusion 63
References 65
ABSTRACT IN KOREAN 72
Table 4.1. Hyperparameters used in the Transformer Single Model 49
Table 4.2.Comparison of the performance of TQE(%), BLEU(%), and F1 BERT(%) for various ensemble methods and single model (avg) in Spanish-English machine translation 52
Table 4.3. Comparison of the performance of n-gram BLEU(%) for various ensemble methods and single model (avg) in Spanish-English machine translation 53
Table 4.4. Comparison of performance of ROUGE-L score(%) for various ensemble methods and single model(avg) in Spanish-English machine translation 53
Table 4.5. Performance of TQE (%), BLEU (%), and F1 BERT (%) for single model in Spanish-English machine translation 54
Table 4.6. Example of English-Spanish Neural Machine Translation Results for Various Ensemble Methods 56
Table 4.7. Comparison of performance of Accuracy for various ensemble methods and single model in String Arithmetic 58
Table 4.8. 1000 epoch 58
Table 4.9. Comparison of the performance of TQE (%), BLEU (%), and F1 BERT (%) scores for the various ensemble methods and single model in German-English... 61
Table 4.10. Comparison of the performance of n-gram BLEU (%) for various ensemble methods and single model (avg) in 61
Table 4.11. Comparison of performance the of ROUGE-L score (%) for various ensemble methods and single model (avg) in 62
Figure 3.1. Overall Structure of Traditional Majority in Seq2Seq 28
Figure 3.2. A toy example: Traditional Majority 29
Figure 3.3. Overall Structure of Survival in Seq2Seq 30
Figure 3.4. A toy example: Recurrent Ensemble using Survival 32
Figure 3.5. Overall Structure of Consensus in Seq2Seq 33
Figure 3.6. A toy example: Recurrent Ensemble using Consensus 35
Figure 3.7. Overall Structure of Traditional Majority in Transformer 36
Figure 3.8. A toy example: Traditional Majority 39
Figure 3.9. Overall Structure of Survival in Transformer 40
Figure 3.10. A toy example: Recurrent Ensemble using Survival 42
Figure 3.11. Overall Structure of Consensus in Transformer 43
Figure 3.12. A toy example: Recurrent Ensemble using Consensus 45
Figure 4.1. TQE evaluated during Training 55
Figure 4.2. Accuracy evaluated during Training 59