Title Page
Contents
ABSTRACT 10
Ⅰ. Introduction 12
Ⅱ. Related Work 16
2.1. Transformer 16
2.2. Positional Encoding 19
2.2.1. Absolute Positional Encoding 20
2.2.2. Relative Positional Encoding 21
2.3. Sequence prediction considering time intervals 22
2.4. Sequence Similarity Measure 24
2.4.1. Dynamic Time Warping (DTW) 24
2.4.2. Soft Dynamic Time Warping (Soft-DTW) 27
Ⅲ. Methodology 29
3.1. Time-Positional Encoding 30
3.2. Output Layer 33
3.3. Loss function 35
Ⅳ. Experiment 42
4.1. Datasets 43
4.1.1. Synthetic Dataset 43
4.1.2. Real-world Datasets 44
4.2. Comparison Methods and Evaluations 45
4.3. Performance Comparison 47
4.4. Influence of Hyperparameters 49
4.4.1. Influence of the Sequence Length 49
4.4.2. Influence of the number of Sequence 51
4.4.3. Influence of Time Interval Range 53
4.4.4. Influence of Time Parameter 55
Ⅴ. Conclusion 57
REFERENCES 59
ABSTRACT IN KOREAN 65
Table 1. Real-world dataset statistics. 44
Table 2. Results on one synthetic dataset and two real-world datasets. 47
Table 3. Results on variations in sequence length. 50
Table 4. Results on variations in the number of sequences. 52
Table 5. Results on variations in the time interval range. 54
Figure 1. The architecture of Transformer 18
Figure 2. A sine function that varies depending on a cycle. 20
Figure 3. Comparison between Euclidean distance and DTW. (a) represents the matching using Euclidean distance, while figure (b)... 24
Figure 4. DTW matrix with the first data of length 4 and the second data of length 5. The optimal path is represented by the yellow ellipse. 25
Figure 5. Soft-DTW matrix with the first data of length 4 and the second data of length 5. 28
Figure 6. Overall structure of the proposed method 34
Figure 7. Time Soft-DTW matrix 38
Figure 8. Influence of Hyperparameters 56