Work

Research on method to detect reduplicative Chinese short texts

Year: 2014
Type: article
Abstract: The article presents an effective algorithm framework for text de-duplication, focusing on redundancy problem of Chinese short texts. In view of the brevity and huge volumes of short texts, Bloom Filter have been introduced, Trie tree and the SimHash algorithm have been introduced. In the first stage of the algorithm framework, Bloom Filter or Trie tree is designed to remove duplications completely;in the second stage, the SimHash algorithm is used to detect similar duplications. This text has designed the parameters used in the algorithm framework, and the feasibility and rationality is testified.
Author: Gao Xian
Language: English
Open Access status: diamond
OpenAlex ID: W2374530766
FWCI:
Cites:
Cited by: 1
Related to: 10