The impact of pre-processing and feature selection on text classification
Nowadays text classification dealing with unstructured and high-dimensionality text document. These textual data can be easily retrieved from social media platform. However, those textual data are hard to managed and processed for classification purposes. Pre-processing activities and feature select...
| Main Authors: | , |
|---|---|
| Format: | Conference or Workshop Item |
| Language: | English English |
| Published: |
2019
|
| Subjects: | |
| Online Access: | http://umpir.ump.edu.my/id/eprint/25395/ http://umpir.ump.edu.my/id/eprint/25395/1/17.%20The%20impact%20of%20pre-processing%20and%20feature%20selection.pdf http://umpir.ump.edu.my/id/eprint/25395/2/17.1%20The%20impact%20of%20pre-processing%20and%20feature%20selection.pdf |
| Summary: | Nowadays text classification dealing with unstructured and high-dimensionality text document. These textual data can be easily retrieved from social media platform. However, those textual data are hard to managed and processed for classification purposes. Pre-processing activities and feature selection are two methods to process the text document. Therefore, this paper is presented to evaluate the effect of pre-processing and feature selection on the text classification performance. A tweet dataset is utilized and pre-processed using several combinations of pre-processing activities (tokenization, removing stopwords and stemming). Later, two feature selection techniques (Bag-of-Words and Term FrequencyInverse Document Frequency) are applied on the pre-processed text. Finally, Support Vector Machine classifier are used to test the classification performances. The experimental results reveal that the combination of pre-processing technique and TF-IDF approach achieved greater classification performances compared to BoW approach. Better classification performances hit when the number of features is decreased. However, it is depending on the number of features obtained from the preprocessing activities and feature selection technique chose. |
|---|