TY - GEN
T1 - A minimal variance estimator for the cardinality of big data set intersection
AU - Cohen, Reuven
AU - Katzir, Liran
AU - Yehezkel, Aviv
N1 - Publisher Copyright: © 2017 ACM.
PY - 2017/8/13
Y1 - 2017/8/13
N2 - In recent years there has been a growing interest in developing "streaming algorithms" for efficient processing and querying of continuous data streams. These algorithms seek to provide accurate results while minimizing the required storage and the processing time, at the price of a small inaccuracy in their output. A fundamental query of interest is the intersection size of two big data streams. This problem arises in many different application areas, such as network monitoring, database systems, data integration and information retrieval. In this paper we develop a new algorithm for this problem, based on the Maximum Likelihood (ML) method. We show that this algorithm outperforms all known schemes in terms of the estimation's quality (lower variance) and that it asymptotically achieves the optimal variance.
AB - In recent years there has been a growing interest in developing "streaming algorithms" for efficient processing and querying of continuous data streams. These algorithms seek to provide accurate results while minimizing the required storage and the processing time, at the price of a small inaccuracy in their output. A fundamental query of interest is the intersection size of two big data streams. This problem arises in many different application areas, such as network monitoring, database systems, data integration and information retrieval. In this paper we develop a new algorithm for this problem, based on the Maximum Likelihood (ML) method. We show that this algorithm outperforms all known schemes in terms of the estimation's quality (lower variance) and that it asymptotically achieves the optimal variance.
KW - Cardinality estimation
KW - Data mining
KW - Set intersection
KW - Streaming algorithms
UR - http://www.scopus.com/inward/record.url?scp=85029104602&partnerID=8YFLogxK
U2 - https://doi.org/10.1145/3097983.3097999
DO - https://doi.org/10.1145/3097983.3097999
M3 - منشور من مؤتمر
T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
SP - 95
EP - 103
BT - KDD 2017 - Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
T2 - 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2017
Y2 - 13 August 2017 through 17 August 2017
ER -