Multilevel Association Rule Mining in Distributed Environment

Abstract

Multilevel association rule discovers knowledge from conceptual hierarchical data set and thus provides more significant information than single level association rule. However, existing multilevel association rule mining algorithms have limitation of processing speed while analyzing big data. To overcome this, Hadoop-based distributed multilevel association rule mining approach is proposed which process the transactional dataset into partitions then transfers each task to all participating nodes. Thus, it reduces inter node message passing in the cluster. The proposed methodology is applied in two phases. In the first phase, the transactional dataset is generated from big sales dataset using Hadoop MapReduce framework. Then, a proposed distributed multilevel frequent pattern mining algorithms MR-MLAB (MapReduce based Multilevel Apriori using Bottom-up Approach) and MR-MLAT (MapReduce based Multilevel Apriori using Top-down Approach) are used to generate level-crossing frequent itemset for each level of concept hierarchy. Performance of the system is compared based on minimum support threshold at different level of concept hierarchy and also by varying dataset size. Moreover, time efficiency of proposed algorithms is compared with existing Traditional Multilevel Apriori (TMLA) algorithm. Due to ancestor relationship, this proposed distributed multilevel frequent pattern mining algorithm generates huge amount of hierarchical redundancy. Thus, to improve the performance of the system, such hierarchical redundancy needs to be eliminated. In second phase, distributed multilevel frequent pattern mining algorithm is applied on regional transactional dataset to generate frequent k-itemsets for each region. Then, multilevel association rules are generated for each region. These generated regional multilevel rules are so large that it becomes complex to analyze it using traditional methods. Hence, MR-MCIRD (MapReduce based Multilevel Consistent and Inconsistent Rule Detection) algorithm is proposed to derive consistent and inconsistent multilevel rules for each region. Further, extracted multilevel consistent and inconsistent rules are evaluated and compared based on confidence, all-confidence, cosine, interestingness of a rule, lift and conviction interestingness measures. The proposed methodology helps organization to improve marketing strategy for regions where inconsistent rules are relatively higher than consistent rules. Pruned interesting rules also provide useful and actionable knowledge to the users.