Farklı veri kümeleri üzerinde metin madenciliği tabanlı metin temsili ve sınıflandırma yöntemlerinin performansının incelenmesi
Year: 2026
Type: dissertation
Abstract: The main objective of this thesis is to automatically extract information from fault descriptions recorded in Maintenance, Repair and Overhaul (MRO) software systems and to assign labels such as Fault Type, Fault Criticality, Fault Source and Task Code using text mining and machine learning techniques. Since the manual labeling of large-scale and unstructured fault records is a time-consuming and labor-intensive process, this study investigates the effectiveness of automated classification approaches in order to improve efficiency and support decision-making mechanisms in maintenance processes. Within the scope of this study, two different datasets were utilized. The first dataset consists of fault descriptions obtained from an MRO software system. The second dataset contains news articles and their corresponding category labels and was used to evaluate the generalizability and applicability of the proposed approach in a different domain. By using these two datasets, it was possible to analyze whether the developed methodology is domain-specific or can be successfully applied to different types of textual data. Before the classification stage, a preprocessing was applied to both datasets. This process included the removal of punctuation marks and special characters; conversion of all text to lowercase, elimination of stop words with low semantic contribution; and the application of lemmatization techniques with Zeyrek library to reduce words to their root or base forms. These preprocessing steps helped to improve text consistency and ensured that the data were more suitable for machine learning algorithms. Following preprocessing, various text representation and feature extraction methods were employed to convert textual data into numerical form. Both traditional statistical approaches and modern representation techniques were evaluated in this study. The text representation methods used include Bag of Words (BoW), TF-IDF (Term Frequency–Inverse Document Frequency), Word2Vec, Doc2Vec, FastText, BERT (Bidirectional Encoder Representations from Transformers), and Latent Dirichlet Allocation (LDA). While BoW and TF-IDF provide simple yet effective representations based on word frequencies, Word2Vec, Doc2Vec, and FastText aim to capture semantic relationships between words in a vector space. BERT offers contextualized word representations by considering the usage context of words within sentences, whereas LDA is a probabilistic topic modeling technique designed to identify latent topics in text corpora. The extracted feature representations were then used to train various machine learning–based classification algorithms. The classifiers evaluated in this study include Naive Bayes, Decision Trees, Random Forest, Support Vector Machines (SVM), K-Nearest Neighbor (KNN), XGBoost, and Voting Classifier. These algorithms were combined with different text representation methods, resulting in a large number of experimental configurations. Model performance was evaluated and compared using standard metrics such as accuracy, precision, recall, and F1 score. To further validate the reliability and stability of the classifiers, 5-fold cross-validation was applied for each experimental configuration. This approach involved partitioning the dataset into five equal subsets, training the model on four folds and testing on the remaining fold in each iteration. The process was repeated so that each fold served as a test set once, allowing performance metrics to be averaged across folds. This methodology ensured that the evaluation accounted for variability in data splits and provided a more robust estimate of model generalizability across different text representations and classification algorithms. Overall experimental results indicate that traditional text representation methods, particularly Bag of Words and TF-IDF, achieved the highest and most consistent performance when combined with Support Vector Machines (SVM) for both datasets. In the fault dataset, TF-IDF and BoW–based SVM models achieved accuracy levels of approximately 93% for Fault Criticality, Fault Source, and Operation Type, while the accuracy for Fault Type classification was around 91%. Random Forest and XGBoost algorithms also demonstrated strong performance with these representations, yielding accuracy values in the range of 91–92%. The results obtained from the news dataset further support these findings. Using TF-IDF features, the SVM classifier achieved approximately 92% accuracy in the news classification task, while Voting Classifier and XGBoost models reached accuracy levels of 91% and 89%, respectively. When BoW representations were used, Naive Bayes and Voting Classifier achieved accuracies of approximately 91%, whereas SVM showed slightly lower performance compared to TF-IDF, with an accuracy of 87%. These results suggest that TF-IDF is more effective in capturing discriminative word importance in news texts. Distributed word embedding methods such as Word2Vec and FastText exhibited moderate performance across both datasets. In the fault dataset, accuracy values obtained with these methods generally ranged between 70% and 86%, while in the news dataset, accuracy values varied between 83% and 88%. Although these approaches were competitive in certain scenarios, they did not outperform traditional frequency-based representations. Doc2Vec yielded the lowest performance among all representation methods, with accuracy values ranging from 32% to 69% in the fault dataset and not exceeding 84% in the news dataset, indicating limited discriminative capability for the given classification tasks. The contextual language model BERT demonstrated stable but comparatively lower performance than BoW and TF-IDF. In the fault dataset, BERT-based SVM models achieved accuracy values between 73% and 79%, whereas in the news dataset, accuracy increased to approximately 85–87%. The relatively better performance on the news dataset can be attributed to the more general linguistic structure of news texts. However, limited data size and the absence of extensive fine-tuning restricted the full potential of BERT in this study. Topic modeling–based LDA produced moderate results for both datasets, with accuracy values ranging between 63% and 83% for the fault dataset and 80% to 85% for the news dataset. While LDA was able to capture general thematic structures, it did not provide sufficiently discriminative features for direct classification when compared to frequency-based approaches. In conclusion, the experimental results demonstrate that large-scale and unstructured fault records, as well as text data from different domains such as news articles, can be effectively classified using automated text mining and machine learning techniques. The findings show that TF-IDF and Bag of Words representations combined with Support Vector Machines deliver over 90% accuracy, high F1 scores, and stable performance across different datasets.
Author: Bekar, Abdulkerim
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162181817
